{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation from Scratch in Python\n",
    "\n",
    "**Training a Neural Network to mimic the sine function using pure NumPy**\n",
    "\n",
    "This jupyter notebook intends to be a hands-on tutorial for the implementation of a simple MLP with backpropagation. The neural network is trained to mimic the sin function from data. Special focus is on the backward pass which is just a special case of reverse-mode automatic differentiation\n",
    "\n",
    "Simplifications:\n",
    "- No train-val-test split\n",
    "- No regularization\n",
    "- No Stochastic Minibatching\n",
    "- Simple Gradient Descent with constant learning rate\n",
    "- No training modifications\n",
    "\n",
    "---\n",
    "\n",
    "Given:\n",
    "\n",
    "- $x \\in \\Re^{N \\times 1}$ and $y \\in \\Re^{N \\times 1}$ (N pairs of (x, y) samples that somehow follow a sine curve)\n",
    "- weight matrices and biases with the correct shapes\n",
    "- sigmoid activation function $\\sigma(x) = \\frac{1}{1 + e^{-x}}$\n",
    "\n",
    "-> The sigmoid activation function has a nice property that its derivative can\n",
    "expressed using the primal output\n",
    "\n",
    "$$\n",
    "\\sigma ' (x) = \\sigma(x) (1 - \\sigma(x))\n",
    "$$\n",
    "\n",
    "Notice the convention of the data matrices being batch x spatial. As such, weight matrices have to be right-multiplied in order to affect the spatial dimensions.\n",
    "\n",
    "---\n",
    "\n",
    "### Forward Pass\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "y_1 &= x \\cdot W_1\n",
    "\\\\\n",
    "\\tilde{y}_1 &= y_1 + b_1\n",
    "\\\\\n",
    "\\hat{y}_1 &= \\sigma(\\tilde{y}_1)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "y_2 &= \\hat{y}_1 \\cdot W_2\n",
    "\\\\\n",
    "\\tilde{y}_2 &= y_2 + b_2\n",
    "\\\\\n",
    "\\hat{y}_2 &= \\sigma(\\tilde{y}_2)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "y_3 &= \\hat{y}_2 \\cdot W_3\n",
    "\\\\\n",
    "\\tilde{y}_3 &= y_3 + b_3\n",
    "\\\\\n",
    "\\hat{y}_3 &= \\sigma(\\tilde{y}_3)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "y_4 &= \\hat{y}_3 \\cdot W_4\n",
    "\\\\\n",
    "\\tilde{y}_4 &= y_4 + b_4\n",
    "\\\\\n",
    "\\hat{y}_4 &= I(\\tilde{y}_3)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "L = \\frac{1}{2} \\text{mean\\_over\\_all\\_axes}((\\hat{y}_4 - y)^2)\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Weight Initiliazation\n",
    "\n",
    "(Xavier Glorot Uniform init)\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "W_{i,kl} &\\propto \\mathcal{U}(-lim, lim)\n",
    "\\\\\n",
    "lim &= \\sqrt{\\frac{6}{\\text{fan\\_in}_i + \\text{fan\\_out}_i}}\n",
    "\\\\\n",
    "b_{i,k} &= 0\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "(Run Forward Pass and save all **activated layer states**)\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\bar{L} &= 1.0\n",
    "\\\\\n",
    "\\bar{\\hat{y}}_4 &= \\bar{L} \\frac{\\hat{y}_4 - y}{N}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\bar{\\tilde{y}}_4 &= \\bar{\\hat{y}_4} \\odot 1\n",
    "\\\\\n",
    "\\bar{b}_4 &= \\text{sum\\_over\\_batch}(\\bar{\\tilde{y}}_4)\n",
    "\\\\\n",
    "\\bar{y}_4 &= \\bar{\\tilde{y}}_4\n",
    "\\\\\n",
    "\\bar{\\hat{y}}_3 &= \\bar{y}_4 \\cdot W_4^T\n",
    "\\\\\n",
    "\\bar{W}_4 &= \\bar{y}_4 \\cdot \\hat{y}_3^T%\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\bar{\\tilde{y}}_3 &= \\bar{\\hat{y}_3} \\odot \\hat{y}_3 \\odot (1 - \\hat{y}_3)\n",
    "\\\\\n",
    "\\bar{b}_3 &= \\text{sum\\_over\\_batch}(\\bar{\\tilde{y}}_3)\n",
    "\\\\\n",
    "\\bar{y}_3 &= \\bar{\\tilde{y}}_3\n",
    "\\\\\n",
    "\\bar{\\hat{y}}_2 &= \\bar{y}_3 \\cdot W_3^T\n",
    "\\\\\n",
    "\\bar{W}_3 &= \\bar{y}_3 \\cdot \\hat{y}_2^T\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\bar{\\tilde{y}}_2 &= \\bar{\\hat{y}_2} \\odot \\hat{y}_2 \\odot (1 - \\hat{y}_2)\n",
    "\\\\\n",
    "\\bar{b}_2 &= \\text{sum\\_over\\_batch}(\\bar{\\tilde{y}}_2)\n",
    "\\\\\n",
    "\\bar{y}_2 &= \\bar{\\tilde{y}}_2\n",
    "\\\\\n",
    "\\bar{\\hat{y}}_1 &= \\bar{y}_2 \\cdot W_2^T\n",
    "\\\\\n",
    "\\bar{W}_2 &= \\bar{y}_2 \\cdot \\hat{y}_1^T\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\bar{\\tilde{y}}_1 &= \\bar{\\hat{y}_1} \\odot \\hat{y}_1 \\odot (1 - \\hat{y}_1)\n",
    "\\\\\n",
    "\\bar{b}_1 &= \\text{sum\\_over\\_batch}(\\bar{\\tilde{y}}_1)\n",
    "\\\\\n",
    "\\bar{y}_1 &= \\bar{\\tilde{y}}_1\n",
    "\\\\\n",
    "&\\left[ \\bar{x} = \\bar{y}_1 \\cdot W_1^T \\right]\n",
    "\\\\\n",
    "\\bar{W}_1 &= \\bar{y}_1 \\cdot x^T\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Learning:\n",
    "\n",
    "Approximately solve a non-convex optimization problem in the parameter space from a randomized initial estimate.\n",
    "\n",
    "1. Initialize weights and biases\n",
    "2. Run forward **and** backward pass (here for all samples together - no minibatching)\n",
    "3. Update each parameter with its gradient, e.g., $W_1 \\leftarrow W_1 - \\eta \\bar{W}_1$\n",
    "4. Repeat until loss is sufficiently decreased"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More details on the backward pass:\n",
    "\n",
    "We used the following primitive pullback/vJp/backprop/adjoint(¹) rules\n",
    "\n",
    "1. Pullback from loss cotangent to guess cotangent ($\\bar{L} \\rightarrow \\bar{\\hat{y}}_4$)\n",
    "2. Pullback over broacasted function activation ($\\bar{\\hat{y}}_i \\rightarrow \\bar{\\tilde{y}}_i$)\n",
    "3. Pullback over batch broadcast vector addition ($\\bar{\\tilde{y}}_i \\rightarrow \\bar{b}_i$)\n",
    "4. Pullback over addition ($\\bar{\\tilde{y}} \\rightarrow \\bar{y}_i$)\n",
    "5. Pullback over matrix-vector (or here matrix-matrix) multiplication ($\\bar{y}_i \\rightarrow \\bar{\\hat{y}}_{i-1}$)\n",
    "6. Pullback over matrix-vector (or here matrix-matrix) multiplication into the matrix ($\\bar{y}_i \\rightarrow \\bar{W}_i$)\n",
    "\n",
    "Operation (3) and (6) map back into the parameter space. Hence, those are the final ones, giving gradients. The other operations let the *cotangents flow reversely over the computational graph*.\n",
    "\n",
    "If you want more details, on why these rules are the way they are, here are more interesting videos on their derivation:\n",
    "\n",
    "\n",
    "1. Loss to guess\n",
    "    1. Pullback of L2 Loss: https://youtu.be/TonUAqYCWAY\n",
    "    2. Pullback of a (mean) aggregator, similar to scalar multiplication: https://youtu.be/ho8v1FpoaEg\n",
    "2. Pullback over broadcasted function: https://youtu.be/bLE6xsVSTUs\n",
    "3. Over addition\n",
    "    1. Pullback over scalar addition (holds also for vector and matrix addition): https://youtu.be/SY2ga4ylwVM\n",
    "    2. Pullback over a scather operation: The reverse will just be the sum the cotangents over the scattered xis\n",
    "4. Pullback over scalar addition (holds also for vector and matrix addition):https://youtu.be/SY2ga4ylwVM\n",
    "5. Over Matrix-Vector product\n",
    "    1. Pullback over a matrix-vector product: https://youtu.be/lqIhocjJLUc\n",
    "    2. (More precisely we do a matrix-matrix product to account for batching): https://youtu.be/O5YealZxi68\n",
    "6. Over Matrix-Vector product\n",
    "    1. Pullback over a matrix-vector product: https://youtu.be/lqIhocjJLUc\n",
    "    2. (More precisely we do a matrix-matrix product to account for batching): https://youtu.be/O5YealZxi68\n",
    "\n",
    "\n",
    "(¹): The idea of how NNs obtain their parameter gradients in a backward pass is\n",
    "a concept found in many disciplines of science. Hence, the history is filled\n",
    "with multiple re-inventions\n",
    "(https://en.wikipedia.org/wiki/Backpropagation#History) and namings vary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_SAMPLES = 200\n",
    "LAYERS = [1, 10, 10, 10, 1]\n",
    "LEARNING_RATE = 0.1\n",
    "N_EPOCHS = 30_000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python convention is batch x spatial\n",
    "x_samples = np.random.uniform(low=0.0, high=2 * np.pi, size=(N_SAMPLES, 1))\n",
    "y_samples = np.sin(x_samples) + np.random.normal(loc=0.0, scale=0.3, size=(N_SAMPLES, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f1cde151870>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKU0lEQVR4nO3df3RU1bk38O8kkAmkSSAESIAAEagQAwShYIRbFWMBuQq3rl5FfQVr6auFe1Fsq+lSKZe2kdpWqPCK4g+0FtHeFn/RxgJBuNBQFBqvgCKkQShmoASYIQESmsz7B504mcw5Z58z58c+Z76ftbKWSc7MbIck+zl7P8+zfeFwOAwiIiIil0hxegBEREREejB4ISIiIldh8EJERESuwuCFiIiIXIXBCxEREbkKgxciIiJyFQYvRERE5CoMXoiIiMhVujg9ALO1tbXh888/R2ZmJnw+n9PDISIiIgHhcBhnz55Fv379kJKivrbiueDl888/R0FBgdPDICIiIgOOHj2KAQMGqF7jueAlMzMTwKX/+aysLIdHQ0RERCJCoRAKCgra53E1ngteIltFWVlZDF6IiIhcRiTlgwm7RERE5CoMXoiIiMhVGLwQERGRqzB4ISIiIldh8EJERESuwuCFiIiIXIXBCxEREbkKgxciIiJyFc81qaPk1toWxq66Uzhx9gL6ZKZjfGEOUlN4xhURkZdYuvKybds23HTTTejXrx98Ph/eeOMN1evfe+89+Hy+Th+BQMDKYZJHVO6tx6SlVZi1eicWrKvBrNU7MWlpFSr31js9NCIiMpGlwUtTUxNGjx6NlStX6nrcgQMHUF9f3/7Rp08fi0ZIXlG5tx73vbIH9cELHb4eCF7Afa/sYQBDROQhlm4bTZs2DdOmTdP9uD59+qBHjx7mD4g8qbUtjMVv70c4zvfCAHwAFr+9HzcU5XELiYjIA6RM2C0pKUF+fj5uuOEG7NixQ/Xa5uZmhEKhDh+UXHbVneq04hItDKA+eAG76k7ZNygiIrKMVMFLfn4+Vq1ahd/+9rf47W9/i4KCAlx77bXYs2eP4mMqKiqQnZ3d/lFQUGDjiEkGJ84qBy5GriMiIrlJVW10+eWX4/LLL2///Oqrr0ZtbS2efPJJ/OpXv4r7mPLycixcuLD981AoxAAmyfTJTDf1OiIikptUwUs848ePx/bt2xW/7/f74ff7bRwRyWZ8YQ7ys9MRCF6Im/fiA5CXfalsmoiI3E+qbaN4ampqkJ+f7/QwSGKpKT4suqkIwKVAJVrk80U3FTFZl4jIIyxdeWlsbMShQ4faP6+rq0NNTQ1ycnIwcOBAlJeX49ixY3j55ZcBAMuWLUNhYSGuuOIKXLhwAc899xyqqqrwxz/+0cphkgdMLc7H03deicVv7++QvJuXnY5FNxVharF1ATAb4xER2cvS4OWDDz7Adddd1/55JDdl9uzZWLNmDerr63HkyJH277e0tODBBx/EsWPH0L17d4waNQqbNm3q8BxEEbFBww1FebihKM/WQKJyb32ngCnfhoCJiCiZ+cLhcLw0AdcKhULIzs5GMBhEVlaW08Mhi8gQNEQa48X+AkVCpafvvJIBDBGRID3zt/Q5L0SxZOimq9UYD7jUGK+1zVP3BkREUmDwQq4iS9DAxnhERM5h8EKuIkvQwMZ4RETOkb7PC1E0WYIG0YZ3uV/yo7q2gZVIREQmYvBCriJLN93xhTnIy/IjEGqO+30fgB7du+LB12s6XMNKJCKixHHbiFwl0k1Xae3Ch0sBgtXddDfuD+DCP9oUxxAGcPrcxU7BjZ1JxUREXsXghVxFhm66kWqnM+cuxv1+dveu6NG9a9zvsRKJiChxDF7IdSLddPOyO24N5WWnW95bRa3aKSLF51MMbABWIhERJYo5L+RKU4vzbe+mC2hXOwHAqaYWoediJRIRkTEMXkhaWmcGpab4UDqkl61jMjPgsDqpmIjIqxi8kJRkaP8fj2jAkZPRFaebLsbdXvLh0haX1UnFRERexZwXko4M7f+ViFY7/WhGcfvnsd8HrE8qJiLyMgYvJBVZ2v8rEa12unFUP8eSiomIvI7bRiQVPe3/7c53iYhUO8Vua+XFbGtNLc7H5OF98avqw/js1DkMyumO/1M6GGldeM9ARJQIBi8kFVna/2sRqXaKl7fz3PY6x/N2iIjcjsELSUWW9v8i1KqdInk7sZtbkbwdbh0RERnH9WuSSiQhVo0d7f8TIXveDhGR2zF4Iamkpvhw82j1FYmbR+dLXamjJ2+HiIj0Y/BCUmltC+OtD9VLod/6sF7qVQu35O0QEbkVgxeSikj7fdlXLdyUt0NE5EYMXkgqXli1EG1kJ3PeDhGRzBi8kFS8sGoh2shO5rwdIiKZMXghqXhl1SLSyI4ddomIzMc+LySVyKrFfa/sgQ/oUG7stlULkUZ2RESkny8cDstbtmFAKBRCdnY2gsEgsrKynB4OGSTjqdKtbWEGIkREFtEzf3PlhaQk26qFjMEUEVGy4soLkQalVv+RMIo5LEREidMzfzNhl0gFW/0TEcmHwQuRikRa/be2hVFd24A3a46huraBAQ4RkUmY80KkwmjTPObIEBFZhysvZCu3rUbkfsmv+7pIjkzsik0geAH3vbIHlXvVz24iIiJ1XHkh27hyNUI0tvrndVo5Mj5cypG5oSiPZdZERAZx5YVs4dbViJNNzbquSyRHhoiIxDB4oYRpbQW5uWJH71lLXjhYkohIdtw2ooSIbAXpWY0oHdLL6iHrEjlrKRC8EDf48uHSeUWRs5YSPViSXXyJiLQxeCHDlJq3RbaCIs3b3LwaofesJb3BTjRX5gQRETmA20ZkiJ6toERXI5ym54ToSLADoNPJ2GoHS7o1J4iIyAlceSFD9GwFJbIaIQs9Zy1Fgp3YVZQ8hVUUVigREenD4MVhbs1x0LMVpHfrJUK29yY1xSeck6Mn2HFzThARkRMYvDjIzTkOereC9K5GuPm9iRANdkQDwUDoAqprG6QJ5oiInGLpqdLbtm3DE088gd27d6O+vh7r16/HzJkzVR/z3nvvYeHChdi3bx8KCgrwyCOPYM6cOcKv6ZZTpd1+UnFrWxiTllZpbgVtf2hyhwlWZDXF7e+NXtW1DZi1eqfmdTkZXXGq6WL7524L5oiI1EhzqnRTUxNGjx6NlStXCl1fV1eH6dOn47rrrkNNTQ3uv/9+fOtb38K7775r5TBt5+a+JxFGE1MjqxEzSvqjdEivuFtFbnlvzDrqIJITpLWGEh24AEzmJaLkZem20bRp0zBt2jTh61etWoXCwkL8/Oc/BwCMGDEC27dvx5NPPokpU6ZYNUzbeSXHQe9WkAi3vDdmbmtp5QQphURM5iWiZCVVzkt1dTXKyso6fG3KlCm4//77nRmQRdzc9ySWnsRUEW54b0T72+ihFAjmZKShoalF8XGyBHNERHaSKngJBALo27dvh6/17dsXoVAI58+fR7du3To9prm5Gc3NX5w/EwqFLB9nomTve6K3ykdPFY4WN7w3VpU1xwsEA8HzeOD1DzUf64ZAl4jILFIFL0ZUVFRg8eLFTg9DF5n7nhjZDjGzpFnm9wawflsrNhCsrm0QepysDf6IiKwgVYfdvLw8HD9+vMPXjh8/jqysrLirLgBQXl6OYDDY/nH06FE7hpoQo8muVjPS5bVybz0mLa3CrNU7sWBdDWat3olJS6sMJ5HK+t5E2L2tpZXM68Ol4FLmBn9ERGaTKngpLS3F5s2bO3xt48aNKC0tVXyM3+9HVlZWhw830NNy3g5Gqnysamkv23sTze5tLdmDOSIiJ1i6bdTY2IhDhw61f15XV4eamhrk5ORg4MCBKC8vx7Fjx/Dyyy8DAO69916sWLEC3//+9/HNb34TVVVVeP3117FhwwYrh+kYs5NdE7Hzrw26tkOsbmkv03sTzYltLaVk3p4ZXfFvJf2R3S0NrW1h3f10iIjcytLg5YMPPsB1113X/vnChQsBALNnz8aaNWtQX1+PI0eOtH+/sLAQGzZswAMPPIDly5djwIABeO655zxVJh3LzGRXPaInt8Mnm/DCjjqhx0W2Q+woaXbqvVFj9KiDREUHcxv3B/BGzec41dSC53ccxvM7DnfIS/JCd2IiIjWWdth1gls67Dop3uQm6tW5V6F0SC+8WXMMC9bVaF6//LYSzCjpb2CUcnMqQNDqPvztrxbi2W11SdOdmIi8Q8/87fpqI9JHafLTErsdIntJs9Wc2NYS2apb/T+dA5fo77OhHRF5AYOXJKI2+YmI3g6RvaQZEMv7SCQ3xO5tLZGtOrV1VDa0IyKvYPCSRLQmPyU9unfF418f2WG7wancD1Ei2zpuyw0xq/yaDe2IyO2kKpUmaxmdtFbOip8nIWtJs1oJ972v7MF/vb0PyzcdtKTM20pmbcF5dSuPiJIHV16SiN5JK7L1c5XKFoNsJc0i/Wpe2HFY8fEy54aIbNX5fIDS4dYybOUREZmBKy9JRKtbazQ9Wz+R3I8ZJf1ROqSXoxO+0a2xaNG5IUa0toVRXduAN2uOobq2oUNjv0SINKyb+y+Fl4IYhe+zoR0ReQFXXpKIWp5KrDzB3A/ZmqGZmc9h5LmszqNRalgX/e81ZmBP1e8TEbkd+7wkIaUJ9ravDMTg3O7CQYiMCa/VtQ2YtXqnKc8V6WkjSqsHi5l5QFpBo2xBJRGRFj3zN4OXJNXyjzb8qvowPjt1DoNyuuP/lA5GWhfxXUQ7J2o9WtvCmLS0SjEvREQkN2T7Q5MBQCgIiLyu0pZV9HMyiCAi6oxN6khVvBWT57bXCa+YWH2uUSL0bI3FE50bsnF/QHhlyY7jEoiI6BIm7CYZM06C1jNRO0GphFtEpMwbgK73STQ/xq09VqxKQiYiMoIrL0nErBUT0Ql4x6G/O5ZzEV3CvWl/AM+rlEc/UDYMg3Mz2scJAJOWVul6n7x8XIKMuU1ElNwYvEjE6iRLs7Y2RCfgFVtq2//bickuUsIdPN+iGrxcnpfZYVzVtQ263yc3HJdghFJuU2QFigc9EpETuG0kicq99Zi0tAqzVu/EgnU1mLV6JyYtrTK106tZWxt6+sVEONW5NrLapCSyihK9DWLkfRLpweK2HisiDf9i3zsiIjsweJGAGXkoWlrbwjh5tlno2j6Z6ao5DmoTtRKnJjsj+TlGt4BkPS7BKNlzm4goeXHbyGF2VO7Ey1mIJ7K1cbqpuVPZb+y2j1KzNDWRye7JjQcwcWhvW/JgjKyiJLIFJNtxCYnwehIyEbkXgxeHWV1iq5SzECsytd48Oh/z1v5FKMchdqI+eLwRK7Yc0hzTii21WLGltkNAZFW+j5FVlERPzI7k2rid6Ht3+OQ5i0dCRNQRt40cZuXdrdqqTqy87HSsvP1KvPVhva4ch+hzjSYOzdU1vkhAVPH7/Zbl+2jl5/hwaVUpdhXFa1tARowvzEFell/zunXvH2HeCxHZiisvDjO7xDZ6BePk2WahLZ1Hp4/AnImFCa8CaW23xHs+AHhmW12n75lVzZLIKoqXtoCMSE3xYdb4gXhy00HV69h8j4jsxuDFYSL5FTkZaQgEz6O6tkF18hTNbYmVm+lHaoov4VWgRLvbRjOzU6/IYYZKvLIFZNTg3Ayh65j3QkR2YvDiMK0JPwygoakFD7z+IQDlfimiuS3xRFZ1zFgFMpLIq8TMlvrJvopilJeb7xGRezHnRQJ62tnHK5/Wk9sSLTbfw2h+SKypxfnY/tBkvDr3Ksy/bqjOUXVm1l19dH5O6ZBeDFwEmPUzQURkJgYvkoie8J+8tQQ5GV3jXhcvcVYrVyWeePkeZjZaiwQKD9zwZd0N7WLxrt45Xmy+R0Tux+BFIpEJPy8rHaeaLipeF9sczMjKhFLVjNlVNkYa2kXYcVfPAwe1Kf1M5GSk4e6Jg5HdLY3vGxHZijkvEtKbOCu6MvHo9BHIzfR3yPeI11/F7PwQpTyY/Ox03Dw6H89uq+u05WXHXT0PHBQXe9Dl+ppjaGhqwQs7DuOFHYf5vhGRrRi8SEhvkqRoR9g5Ews7BAJak7eZVTZKAdHG/QFkd/8bzpzruNLUo3tXVHx9pCmTYbwAbeP+AA8c1Ck1xYfg+UsBC983InISgxcJ6W1Pb6SXiROnBceWHatVSJ0+p7xtpke8AC0vKx0X/tFq6ZEMXmTHURZERCKY8yIhI0mSenJVZDgtWKtCKt5pz9GPFclTUTzwMnSh00pPNLccOGh3vg4PaiQiWXDlRVIijdVit0NuKMoTylWx+jwlEUbHIJqnYrR8PJrMjdecyNfhQY1EJAsGLxJTS5xNZPKSYRIyMgY9W11GysdjqeUeWXWQpAgntvwANqwjInkweJFcvPb0iU5eMkxCesegN98ikcArNqcolpNVSk7mnYwd1BMpPkBtdyrFd+k6IiIrMefFZbQmrzCAh3/7EXYcOqmYA6G3a6oVuRV6x6A338Jo4KVVoq2YRxOn87EVnMw72f3ZadXABbgU2Oz+7LTpr01EFI3Bi8uIbIecOX8Rdzz3Z0xaWhV3MtWTEFy5tx6TllZh1uqdWLCuBrNW71R8Xj30JiXr3WYSCY56du+KvCx/h69nd+uK+8uG4YaivE6PkSHR2cktPxm2G4mIAAYvrqNnYqgPXsC9r+zB8k0HO0yorW1hZHe71B21Z0Zah8dEVydZvcqgp0JK7zaTSHBU8fWR2PHw9Xig7Mvo0e3ScQxnzl/Ek5sOxg3QZKi2cXLLT4btRiIigDkvrmNkYnhy06d4dddn+OHNVwBAp3yNnIyu+LeS/igryuvQedeO3ArRbr56e99EnlurYqtybz2WbfpUKH9IhpUHI++DF16biCgagxeX0ZpAlARCzbj3lT1xv3e66SJe2HEYX4kKGuwsp46XlBzvGr2N+AD14EhvgCbDyoPR98Htr01EFI3bRi6TyEGHSuLla8iwyhDL6KGRkeBoRkl/lA7pZShAA/QnGVvF7MMz3fLaREQRXHlxIaXtkETErqTIsMoQj5mHRuoN0GRaeTD78Ey3vDYREcDgRUoiDdAiE8jO2gbMW7sHZ86bcxZQbLWOjPkNIttMIowEaCJ5NHYx631w22sTETF4kYyeBmipKT5MHJaLx28ZqZjPoldstY4MqwxWMRqgceWBiMhZzHmRiNHS5KnF+bhn4uCEXjtevobX8xuMHIAZ/dh4eTRWsvsgRiIiWdmy8rJy5Uo88cQTCAQCGD16NJ566imMHz8+7rVr1qzB3Xff3eFrfr8fFy54u/FVoqXJZUV5eH7HYaHXUltJAYDq2gbdhz26lUzbQGqcPJKAiEg2lgcvr732GhYuXIhVq1ZhwoQJWLZsGaZMmYIDBw6gT58+cR+TlZWFAwcOtH/u83ljolSTaGmySAl1z+5d8eOZxViy4eO4EzUATFpalXQTpOzbQGYexGjlgZJOHlZJRMnF8uDlF7/4BebOndu+mrJq1Sps2LABL7zwAh5++OG4j/H5fMjL69ye3csSLU1Wy1EBLq2uVHx9JKYW52NKcX6nSWbj/oAjJxXLQtYEVDObBVq5esOVISKyk6U5Ly0tLdi9ezfKysq+eMGUFJSVlaG6ulrxcY2NjRg0aBAKCgowY8YM7Nu3T/Ha5uZmhEKhDh9ulGhpslrL//yYHJXYfA0Ajp/ZQ/GZdSSBlUc9OH1YJRElH0tXXk6ePInW1lb07du3w9f79u2LTz75JO5jLr/8crzwwgsYNWoUgsEgfvazn+Hqq6/Gvn37MGDAgE7XV1RUYPHixZaM306JlCbHu+uN1/JfiZ3ddEkfM5oFWnnUg13HSBARRZOu2qi0tBR33XUXSkpKcM011+B3v/sdevfujWeeeSbu9eXl5QgGg+0fR48etXnE5tDqnBsG8Oj0EZ0mAKW73kjL/+D5Fs1JQ8ZuunSJGc0CrTxQUobDKoko+VgavOTm5iI1NRXHjx/v8PXjx48L57R07doVY8aMwaFDh+J+3+/3Iysrq8OHWymVJkcs2fBxhyV4rbteQGy7R9ZuumTOkQSiQeeOQyd1bw0y8CUiJ1gavKSlpWHs2LHYvHlz+9fa2tqwefNmlJaWCj1Ha2srPvroI+TnJ0fS39TifDw6vSju92JzCMy665XlzB7qLJFeNBGiQeeKLYcwaWlVpwBZrbcMA18icoLl1UYLFy7E7NmzMW7cOIwfPx7Lli1DU1NTe/XRXXfdhf79+6OiogIA8F//9V+46qqrMHToUJw5cwZPPPEEPvvsM3zrW9+yeqhSaG0LY8mG/XG/F5tDYNZdbzJ003WzRHvR6DmJPLq6DIBmBZHMx0gQkXdZHrzceuut+Pvf/47HHnsMgUAAJSUlqKysbE/iPXLkCFJSvlgAOn36NObOnYtAIICePXti7Nix+NOf/oSiovirEV6jZzXFzLte0QlShl4eMozBbon0otEqo48WCZDLf/cRTp/rfF5WbOk8A18icoIvHA57qv41FAohOzsbwWDQlvwXsyfSN2uOYcG6Gs3rlt9Wgn8d1Q+TllZp3vVuf2iy8JjU/n9k6OUhwxjcKt57Z0S8nyv+uxBRovTM3wxeEhC/RDkNP5pRjBtHGfuDXV3bgFmrd2pe9+rcq1A6pFd7tREQ/67XrOZySl1ezX4d2cfgdq1tYTy58QBWbKlN+LkiP4PRz51sK2JEZB4987d0pdJuoVSifKqpBd9Zuwc/3rDP0CF6epNn7Tg80ayqJrePwQtSU3yYOLS3Kc8Vm0vlxGGVRJScbDmY0WvUJtKI1f9zGKv/53D756JL6EZyCKw+m0eGJnYyjMEr9CTwqmEFERE5hSsvBmhNpPHoaZVuZDXFyrteGXp5yDAGr9Aqv/YB6NG9K0vniUhaXHkxwMgEqbdVukwnHcvQy0OGMXiJVnUZAFYQEZG0GLwYYHSC1Lu1IctJxzL08pBhDF6jFSAn0luGiMhKDF4MGF+Yg5yMNJxqajH0eLdtbYjk4Tw6fYSlq0TsJ2INtQBZptU/IqJoLJU26Pf/W4/vrN1j6LGxJaZuodTL4+bR+Xjrw3pbenywnwgRkTexz4tNfV4qfr8fz2yrE77eSNM42cT28jjd1IJ5a+3tvcJ+Ivbhe01EdtEzf3PbKAHlNxZh9IAeeOTNvTjV1LmVejSvbG1EbzO0toUxaWmVYu8VPQnKRsdA1uEqFxHJisFLgm4c1Q9TivM7rUYs2eD9REf2XvEupW7GsWcbERE5gcGLCeKtBEwp9n6iI3uveJNWN2OrVtSIiEQxeLFIMmxtsPeKM6zOQxFdUXty46eYODTXk4E5EcmNwQsZxt4r9rMjD0V0pWzFlkNYseWQrtdnAjARmYHHA5BhWm3mAfcnKMtE6TBQPUdPiNC7Uib6+pV76zFpaRVmrd6JBetqMGv1TkxaWmXauIkoeTB4EdTaFjZ0SrTX2XGqNdl7qrbWyeZGXt+uwMtq/DtAJAduGwlgyag6dmK1np2VXWrdjI28vlcSgPl3gEgeXHnR4JU7RqtZeao12V/ZpbSiZuT19QResuLfASK5MHhRYedSPZEaJyq7phbnY/tDk/Hq3KvwnWsvE3pMboa/09fcXlLPvwNE8mHwosILd4zkDVp5KD5c2sIwu7IrsqI2cWhvsQf4OueFxAto4pG1pJ5/B4jkw5wXFW6/YyTvcPpU7ZONzULXVX18HN/9zYcdu0tnpaNH964InrvoypJ6/h0gkg9XXlSwCRvJxMnKLtGf8ed3HO60SnE8dAFn/hm4uLGknn8HiOTDlRcVbMJGsnGqskvkd8HnA+KlfUSClh7du8LfJQWB0BerOG4484t/B4jkw+BFhZ1L9ew8SqKcOHpC63chDCCskq8aBnD63EX8+lsTkOLzuern3OktOyLqjMGLhshSfWx/h54ZXfFvJf2R3S0NrW3hhP5wsX8EuYHS70J2964ovawX/rA3oPkcJxubMaOkv5XDtITS/7sbVo6IvMgXDqvdL7lPKBRCdnY2gsEgsrKyTHveyMrIxv0BvFHzOU41tbR/L5FAI9I/IvYfIRIKsUstyaa1LYwVVQfx4o7DOHP+oq7Hvjr3KlcfWMoVUiLr6Jm/mbArKDXFh+D5Fry443CHwAUw3qiK/SPIjTbuD2DZpoO6AherSrntxmaMRHJg8CLIikCD/SPIbdR+D5QwL4SIzMbgRZAVgQb7R5DbaP0exMNDOonIbEzYFWRFoMH+EeQ2oj/f868bgmF9M9EnMx1jB/XE7s9O482aY8wTISJTMHgRlGigES/Rj/0jyG1Efw8mDu2N0iG9ULm3Htc8sYWVdERkKgYvghIJNNRKodk/gtxEz++BUiVdJMGdW0lEZBRzXgRFGlUB+lqcR/6Ax+YJRP6AA3Cs5TuRXqK/BwBYSUdElmGfF530NJRrbQtj0tIqxQTHyF3q9ocmAwD7R5BraP0eVNc2YNbqnZrP4/a+L0RkHj3zN7eNdNJztoyeCqXSIb34R5xcQ+v3gJV0RGQlBi8GiJ4twz/g5GVqvwdOVdJZ1QGXnXWJ5MLgxUIshaZkdbqpBSkKp0wD1lTSGT0jTCsw4dljRPJh8GICpT9+LIUmtzFjhaFybz3mre1cZRTLzEo6o5VNWoEJK6aI5MTgJUFaf/xYCk1uYcYKg8jxASk+YMWsMaZN+lpHd/hwqbLphqK8TisqaoHJytvHYMmGj3U/LxFZj6XSCdAqg67cW4+pxfkshSbpifwsixA5PqAtDPTM8Bseq97XjHd0h8hZZY+8uZdnj1FSa20Lo7q2AW/WHEN1bYNUrQ248mKQnrs9PRVKRHYzunIRjxNJ6kZeUyTgOdUkdmo2E+7Ji2TP9bJl5WXlypUYPHgw0tPTMWHCBOzatUv1+t/85jcYPnw40tPTMXLkSPz+97+3Y5i66L3bi1RmzCjpj9IhvRi4kDTMPHTUiSR1I69pZsDBhHvyGrNWYq1kefDy2muvYeHChVi0aBH27NmD0aNHY8qUKThx4kTc6//0pz9h1qxZuOeee/CXv/wFM2fOxMyZM7F3716rh6oLy6DJK8z8WY4kqSuF5j5cunszM0ndyGuKBhw5GWm2/r8QOU1kS1WG7tiWBy+/+MUvMHfuXNx9990oKirCqlWr0L17d7zwwgtxr1++fDmmTp2K733vexgxYgSWLFmCK6+8EitWrLB6qLqwDJq8wsyfZaPHaCTCyGuKBjw/mlGs63mJ3M7MlVgrWRq8tLS0YPfu3SgrK/viBVNSUFZWhurq6riPqa6u7nA9AEyZMkXx+ubmZoRCoQ4fdnDiDpPICmb/LFuVpK6WPKj3NUUDnhtHMeGekotbdhUsTdg9efIkWltb0bdv3w5f79u3Lz755JO4jwkEAnGvDwQCca+vqKjA4sWLzRmwDpE/fiyDJrez4mfZ7CR1keRBva8ZCXhinzcvweclkpVIHye37Cq4vtqovLwcCxcubP88FAqhoKDAltcW/eNHJDsrfpZFj9HQoqdRnN7XFAlMeDQAeYFo9ZBbmqtaGrzk5uYiNTUVx48f7/D148ePIy8vL+5j8vLydF3v9/vh95vXM0Iv3pWRV8j4s2xmGbcStYBH9nJRIhF6bwDcsKtgac5LWloaxo4di82bN7d/ra2tDZs3b0ZpaWncx5SWlna4HgA2btyoeL0MWAZNXmHlz7KRhldOJg+6oVyUkpfo75OR6iE3NFe1fNto4cKFmD17NsaNG4fx48dj2bJlaGpqwt133w0AuOuuu9C/f39UVFQAABYsWIBrrrkGP//5zzF9+nSsW7cOH3zwAZ599lmrh2oIl5SJtBldwXAqedCOFR8io/T8Pum5AYhegZRxJTaa5cHLrbfeir///e947LHHEAgEUFJSgsrKyvak3CNHjiAl5YsFoKuvvhpr167FI488gh/84AcYNmwY3njjDRQXF1s9VN24pEykLZHDDZ1KHjT6B58owqobW72/T6KBfSB0AdW1DZ3GK+vPty8cDstzWIEJQqEQsrOzEQwGkZWVZdnrKP0ARX40ZVlaI9LD7D+4rW1hTFpapRgIRJL/tj80Oe7rRB6vlTyo9Hij3qw5hgXrajSvW35bCWaU9DftdckbrLqxNfL7VF3bgFmrd2o+d05G1w5HYjhxI65n/ubBjAa4pQMhkR6Ve+sxaWkVZq3eiQXrajBr9U5MWlqVUG5HojkrTjS9A9xTLkrysTJXysjvk1Yfp4jYs7xkz+1i8GKAWzoQEomy6g+uGTkrTiQPsgklGWH1ja2R3yeRG4B4ZL8Rd32fFye4pQMhkQgrk1PNWsGwO3nQLeWiJBerc6WM/j4p9XHKyUhDQ1OLZeO1EoMXA7ikTF5i5R9cMxte2Z08yCaUpJfVN7aJ/D7FuwEIBM/jgdc/1DVeWSpsGbwY4JYOhEQirPyD6/YVDNnLRQF5JhOy/sY20d+n2BuA6toGXeOVqcKWwYsBbv+DTBTN6j+4bl/BkKlcNDZQOd3UjCUbPpZiMiF7bmzN/H3SM95EWh5YgaXSCZApCiUyyq5yZDtWCLy8ChHv7008bNfgrMgkD8S/sTXr38Wsn3WR8d5QlJdQywNReuZvBi8J8vIfS0oedv3BtVK8yT0nIw0/mlGMG0fJPXYtSne9Sqzqf0NinLqxNTofaY1XtFfMq3OvSmiVUs/8zW2jBMm0pExklNu3dpQm91NNLfjO2j34v38rRPmNRY6MLVFq1WBKZK4SSQZO5EolEjBpjVfGClsGL0QEwB3JqfGITO7PbKvD6AE9cOOofraNyyxa1WBq2K7BOXbe2JqRj6I2XhkrbNmkjojaufGEdNHJ/ZE390rZbEtLIgEI2zV4nx0d32Vs2sjghSiJtLaFUV3bgDdrjqG6tsGVk3ks0cn9VNNFV3a9NhKAsANw8rCj47tTx3So4bYRUZLwanWcnsndjdsokbte0a0jtmtILqI/0zsO/T2hbWDZ8uIYvBAlAdl6NJhpfGEOcjLScEqlzXmEG7dRUlN8eHR6Eb6zdo/Q9W5JsiZziP5Mr9hSi9/uOZbQz4ZMeXHcNiLyOK+fgp6a4sOPZhRrXufmbZSeGWlC1z06fQS2PzSZgUsSGV+Ygx7duwpda8ZJ0bLkxTF4IfK4ZDgF/cZR+fi/Xy1U/L4P7t5GEd0ayM30u/b/kaznhZuVCAYvCfJiAiR5i4w9GqxQfmMR/t/tY5CT0fEuND873dXbYoCcpaokh111p3Dm3EXh671wswIw5yUhXk2AJG9JponvxlH9MKU4X4o9eTPxMFhSYvSmw+03K1x5MSiSABm7HG/GniKRmWTs0WAlWfbkzSRjqSrJwehNh9tvVhi8GOD1BEjyFk583hApVc3L7jjp5Jm4LcZtcPfRujmJ5ZWbFW4bGaAnAZLnipAMZOvRkIzMOMTVylJVboO7U+Tm5L5X9sAHqB6T4aWbFQYvBiRLAiR5i0w9GpKNmYGBFWfmeLkPUDJQujlJ8QHRi2dqNytmBNd2YvBiQDIlQJK38BR0+8keGGhtg/twaRv8hqI8qSezZBfv5mTsoJ7Y/dlpzYDEjatuDF4MYOY/EYlwQ2DAbXDviHdzovVvJntwrYQJuwYwAZKIRLihQSC3wZOXm4tPGLwYZEfmPxG5mxsCA26Du4fZ1WBuCK6VcNsoAUyAJLKGG5IHRcbohsCA2+DuYEVeihuCayUMXhLEBEgic7kheVB0jG4IDNRKbbkNLger8lLcEFwr4bYREUnDDZ2r9YzRLflx3AaXl5V5KaebWjSv6dm9q5Srblx5ISIpuKEyx8gY3dIgkNvgcrKqGqy1LYwlG/ZrXnf63EVs3B+Q5uc0gsELEUnB7pJdI3k1RsfolsCA2+DysSovRetnOZrTNw3xMHghIinYmTxoNK8mkTEyMCAjrMpL0fN7JGOfH+a8EJEUrE4ejJSZLnl7H+41mFfj5gRHcierToW3MtixA4MXIpKCVX+kgUsrLZOWVmHW6p14fsfhuNeoJT9GAp9A8DxyMtIsGSNRPFYlfUd+30TJFpAzeCEiKVj1R1qpOiieeE25ogOfB17/EKeaWhTLno2OkUiNFdVgkd83rZ9UWQNy5rwQkTTMrsxRqw5SE1kiV+qvEY9s1UPkLVYkfUd+3x7+3Uc4c+5ip+/LHJAzeCEiqZj5R1pPRUW0PpnpQoFPhj8VP5o5EnlZclYPJcINXY6TjRVJ35HftxVVB/HijsM4c/6LIEbmgJzBCxFJx6w/0nqTDKM73ooEPk3NrTjScA43j+7nqYneDV2OyTypKT4sKPsy5k8e5pqfYwYvRORZepIMY5fIRQOfZ7fV4tVdnyEQam7/mpsneqta0ZP83FTOz4RdIvIsrQqmaLHJj6KBT1NLa4fABZDrOAM9rGxF70Zmn+JM5rE0eDl16hTuuOMOZGVloUePHrjnnnvQ2Nio+phrr70WPp+vw8e9995r5TCJyKNEKpi+OXEwXp17FbY/NLnToYo9unU19Lpunej1dBD2uugqswXrajBr9U5MWlrluoDUqywNXu644w7s27cPGzduxDvvvINt27bh29/+tubj5s6di/r6+vaPn/70p1YOk4g8TK3MdNWdV+Kxm65A6ZBenfb2U1N8uHviYMOv68aJ3s4uxzKz6oBQruSYx7Kcl48//hiVlZV4//33MW7cOADAU089hRtvvBE/+9nP0K9fP8XHdu/eHXl5eVYNjYiSjNEKpvmTh+HFPx2OW0Yqyk0TPTsIGzt8U6Qyi0nQ5rJs5aW6uho9evRoD1wAoKysDCkpKfjzn/+s+thf//rXyM3NRXFxMcrLy3Hu3DmrhklESSKSjDijpH/clRalxzz+9ZFCOTNK3DTRW9nl2C30bp2JbC9ZtZKTzCwLXgKBAPr06dPha126dEFOTg4CgYDi426//Xa88sor2LJlC8rLy/GrX/0Kd955p+L1zc3NCIVCHT6IiMwS2XaKbaWen52OHt27SjnRG92esKrLsZvo2ToTCUqYBG0N3dtGDz/8MJYuXap6zccff2x4QNE5MSNHjkR+fj6uv/561NbWYsiQIZ2ur6iowOLFiw2/HhGRluhtp0DoAk41NiMnIw1HTp3Hsk2fwgd0mJz0TvRmNoRLdHvC7C7HbiO6Upb7JT+++5sPNbeXMtO7Cq/kuKVMWQa6g5cHH3wQc+bMUb3msssuQ15eHk6cONHh6//4xz9w6tQpXfksEyZMAAAcOnQobvBSXl6OhQsXtn8eCoVQUFAg/PxERCJSU3wInm/BTys/6TAZ9eh+qSIpOi9Gz0RvZi6EWT1arGhF7xaRrbNA8ILiGVZ52elAGEJBSXVtg9Druik3Sga6g5fevXujd+/emteVlpbizJkz2L17N8aOHQsAqKqqQltbW3tAIqKmpgYAkJ8f/xfO7/fD7/cLPx8RkRFKgUHw3EWEATxQNgyDczN0TfRmNoQzkmiqxk0Ny8wU2Tq775U9qitqJ5ua4zw6HrHtIDflRsnAspyXESNGYOrUqZg7dy527dqFHTt2YP78+bjtttvaK42OHTuG4cOHY9euXQCA2tpaLFmyBLt378bhw4fx1ltv4a677sJXv/pVjBo1yqqhEhGpEgkM1r1/FP86qp9wMrAZuRDRuS1rdtSxR4tJRE5xFg02Si/L1WyUmOIDTgsHQwRYfDzAr3/9a8yfPx/XX389UlJScMstt+CXv/xl+/cvXryIAwcOtFcTpaWlYdOmTVi2bBmamppQUFCAW265BY888oiVwyQiUqWnAkV0tSLR54y33SSC2xNitLbORLeXrhrSq30lR0lbGJi39i94OsVneU6RVw7ctDR4ycnJwdq1axW/P3jwYITDX/yzFxQUYOvWrVYOiYhIl9a2MHYcOil0rZ7AYNN+5apLredU2m4Swe0JcWpbZ6LbS6n/DEhW3n4l5r+6B2pFRXq29YzwUq8Znm1ERKQg0sNjxZZDQteLBgaVe+vx/I7Dhp5TbbtJTTL0aLGbyPZSxMETZ1UDF6u39bzWa4anShMRxaF3dUM0MIgEH1oi2w6xz6m13aT0XID3e7Q4QaQyq3JvPZ7cdFDo+azY1jM7mVsGDF6IiGIYWd24eXS+0B9+0eAjjPjBhpHJLVl6tDhFbXtJNFiNsGJbz4qcLacxeCEizzKanGhkdeOtD+vx/akjNJ9fNPj45sTBcYMN0cnt0ekjkJvpd3VSphfo+VmyalvPiwduMnghIk9KJDnRyB9x0TtX0eDjhqL4zTxFq1zmTCxkwCIBPT9LVm3refHATSbsEpHnJJqcaPSPuMhENXZQT+RkdFX8vlZiLc8fchfRn6UHyr5s2baeFw/cZPBCRJ5iRvO38YU56Nld/8K01kRVubce1zyxBaeaLsb9vmjwoafKhZQZPcBSD63AAQDysvyYP3mo6a8d4cWAl9tGROQpZiQnpqb4MLt0MJZtFiuRBi6dcaR25ypSvaQnsTaZzx8yg109T0T6wfzw5iss/3fz2oGbDF6IyFPMSk4s7P0lXa975txFvLs3gJ4ZaZ2CiZZ/tOEH6z9SDVx6ZaRh6/euQ1oX8QXxZD1/KFFmniklIpHAwcyOuF4KeBm8EJGnmJWcaCTvJbaDan52Om4enY/f7D6muFUU0dDUgt2fnWYwYjGnep4YCRysWB3ySsDLnBci8hSzkhNFchVixaZM1Acv4JltdTjV1CL0eDeVqrqVnm1Fs0UChxkl/TUP8PRaR1yzMXghIk8xKzlR7XmscvjkOZteKXm5oeeJGUnnXsfghYg8x6xqHKXnscqyTZ8m/R211dzQ88TJ1SG3YM4LEXmSWcmJkefZWduAeWv34Mx59dyVRDlxxoyZSaF2MjJu0SZ/TvY8ccPqkNMYvBCRZ5mVnJia4sPEYbl4/JaRuO+VPQCg+1RnEU6cMWNXybDZjI5bpHTZ6Z4nblgdchq3jYiIBNm1jWTkjtpIwzW3JoUmOm7Zm/x5sSOu2bjyQkSkQ+x21MmzzViy4WNTX0PvHbWRVQinSoYTZda4Ze554obVIadx5YWISKfoktfcTL9pz2vkjtroKoSRpFA72ulrMTOZVU/pshKr3hPZV4ecxpUXIqIEiK6SPDp9BHIz/Th88hyWbfoUQOJ31ImsQuhNCpUlN0amZFar3xOZV4ecxuCFiCgBotUrcyYWtk86w/pk4JE393boumvkjJlEznHSkxRqdzt9rfGYeZ0atWomu96T2KTzyEpPsgczDF6IiBKgNz+hcm89lmz4uEPgkpORhken679bT2QVQjToGjuoJ655Yos0uTF6S52NloGrrarcUJTnSL6QLKtfMmDOCxFRgkTzE5TyU043tWDeWv3VPYmsQoh2It792WmpGqbp6aBcubcek5ZWYdbqnViwrgazVu/EpKVVmu+zVh7RiqqDtr8nbq0MswqDFyIiE0wtzsf2hybj1blXYfltJXh17lXY/tDk9sDFipbviZbUigRdMuWYRIiM2+hkL/Lv9OKOw0LjNOs94XEBnXHbiIjIJGpN8RLJT1F7vURLarWSQmVtmKY27kQSmUX+nUS7LJv1nljxs+N2DF6IiGxg1QpGZBUiNhdCTwKwWtAlkmOSk5GGQPA8qmsbbE0gVRp3IpO96Pvfo1tXBM9ftOWIARlXv5zG4IWIyAZWrmBYWVKrtrqDf37e0NSCB17/EIAcCaSJTPai7//dEwuxbNOntjSRk3X1y0nMeSEisoEVLd+jG6TtqjuF8YU5CTVcU6LnWAQZEkgTmexF/53mTx5qWxM5HhfQGVdeiIhsYHbLd7vLZqNXdwKhC1jyzr4O5d4RMhwtkMjJ0Xr+nbRWvMw6rZvHBXTmC4fDnkpPDoVCyM7ORjAYRFZWltPDISLqwIygQ6lBWmTqsrppXHVtA2at3ql53atzr3IsgTTyHgHxJ3ut9yjRfycrgkujZ1i5pUOvnvmbwQsRkc0SmVBa28KYtLRKMSE1sqqw/aHJlk1Sb9Ycw4J1NZrXLb+tBDNK+lsyBhGJBhCJNLizKrjUMya3NbXTM39z24iIyAR6JhW16h4tMpTNuiWBNNFEZiP/Tlaf1i06JpmOdLACgxciogTZeYcrQ9lsIjkldkskUDRChuDS6gBKBqw2IiJKgN1t22VY9dDToj/ZyBBc6gmg3IrBCxGRQU60bZelbFb0PKdkY3VwGV0eX13bEPdnS4YAymrcNiIiMsiJLQKZymatbI7nVlZuqYluT8qwOmc1rrwQERkkeue649BJ1TtlvURXPUTu0hMVySmxojmeG1m1paZne1KW1TkrsVSaiMgg0X4n0cxM5FWrcFK6S390ehF6ZqRZ2lSNzE3iNlIen2ifGyewzwuDFyKyQWRSUdoiiMeOyUOpTDaeyIQKQKqeIF4IpMz6fzDaFNDLfV4YvBARJUDpDleNlY3kWtvCmPh4FQIhsS2teIctRn8PsP8uPd6km5ORhpkl/XBDUZ4rA5lEJNIU0E1BIJvUERHZJJJ/EjvZqokk8u6sbUBKis/UiWVF1UHhwCUyFrXv2d0TRGnV6FRTC17YcRgv7Dgs9eqBFRJJwLW7z41duPJCRGSC6Dvcg8fPYsWWWs3H9OjWFWfOf3G4oRln39z7z1Ugs9lxTpFWbkc0H+TM27CC1vakHUdC2EHP/G1ZtdGPf/xjXH311ejevTt69Ogh9JhwOIzHHnsM+fn56NatG8rKynDw4EGrhkhEZJroqpuJQ3sLPSY6cAESa2wX6TljldjKKisqmbRKz2OZ3UNHVqIVTAAsry6ThWXbRi0tLfjGN76B0tJSPP/880KP+elPf4pf/vKXeOmll1BYWIhHH30UU6ZMwf79+5Ge7t56dCJKLlq9PpQksk2jd+LXK3pLwqpEUD1N0+xosy8Tpe3JvKiE69hVKy9vr1kWvCxevBgAsGbNGqHrw+Ewli1bhkceeQQzZswAALz88svo27cv3njjDdx2221WDZWIyFRqjeS0GJ2UreqWGttUzcoD/4w0TbOzS6zTya9KTQE37g94+hDGeKRJ2K2rq0MgEEBZWVn717KzszFhwgRUV1crBi/Nzc1obm5u/zwUClk+ViIiLUp3yrF5Lkr0TspGu6VGB1daHXutPvDPyIqVXV1iZSk7jk3ATYZDGOORpsNuIBAAAPTt27fD1/v27dv+vXgqKiqQnZ3d/lFQUGDpOImIRE0tzsf2hybj1blXYfltJXh17lV4atYYocfmfsmv67W0uqoCQI9uXZCX1fF587LTserOK7EqTsfenIw0rLz9i7t2qw/8U8vtiGWkS6zRPB27D9/UIxkOYYxH18rLww8/jKVLl6pe8/HHH2P48OEJDUqP8vJyLFy4sP3zUCjEAIaIpBF7p7zj4EmxB+rMtRQ58+jxW0bF3XYALk2CU6/oi//e8zecvdAKAGhoasGSDfuRknIpELPjwD+R0nMjbfaNrpzIvrKRDIcwxqMreHnwwQcxZ84c1Wsuu+wyQwPJy8sDABw/fhz5+V/8IB0/fhwlJSWKj/P7/fD79d2hEBE55WRTs/ZFOq6LppXUGZmktbqwRovOm7DrwL/o3I5N+wNYX3MMp5q+2GqL/f/RopWnc3/ZlzE4t3vcPBYnDt/UIxkOYYxHV/DSu3dv9O4tVgKoV2FhIfLy8rB58+b2YCUUCuHPf/4z7rvvPktek4jIblZPNnpOehY5RiB6dWHr966z7MTkWJEVq9IhvfCD6UWGE2W1Vk4A4MlNn7Z/LXY1RvaVDStPsZaZZTkvR44cQU1NDY4cOYLW1lbU1NSgpqYGjY2N7dcMHz4c69evBwD4fD7cf//9+NGPfoS33noLH330Ee666y7069cPM2fOtGqYRES2suPEX5GTntUm9ViR1YXdn5225MRkLYmcXK23hDw2j0X2lQ2rTrGWnWXBy2OPPYYxY8Zg0aJFaGxsxJgxYzBmzBh88MEH7dccOHAAwWCw/fPvf//7+I//+A98+9vfxle+8hU0NjaisrKSPV6IyDNkmWyM9IU5cfZC+9ZUbHJvXna6lCW5eldEIsFcpAGeHcFmotz2b2IGHg9AROQAp0tvRQ/7ixZ9RIDTPU9EiZ7IHE/k/1fp8E2nDq5U4pZ/EyU8mJGISHJ6clPMEDux6SnFjpc34ZYD/4x2Owa+WLURTYR2mlv+TczA4IWIyCF2TTbxVnnysvzo0b0rgucuCk3qWltZst71J9LtODqPxe5gk9QxeCEi8jCliqLjoWbFzrrRRLaynN4C0yLSOyaaUoVOMq1syI45L0RELqW12tHaFu50WF80H4Ds7l2R3iUVgdAX1/TKSMOMkn64oShPc3VBKTiSLR8E6Ph+HT7ZhCc3HVRs6Kdn3LKuOrkNc16IiDxOZLVDpMHamXMX8et7rkRKik/35Ct799lYsSsnl+dlJpzHIvuqk1cxeCEichnRk51Fy4RPNjVjRkl/3eOQpfus0ZWPRPNYrDxhm9QxeCEichE9qx1WN1iToftsoisfRvNY3Lbq5DXSnCpNRETxRZ+GvGZHnfBqh9UN1pzuPuvkac/JepqzLLjyQkQkMa2DE5WcOHtB6KTpRLr5OnmujtMrHzKsOiUzrrwQEUlKaWVBRGS1w8rW8U4edeD0yofTq07JjisvREQS0nNwYrR4qx1WNlhzqvus0ysfyXqasywYvBARScjIwYlqqx1WNlhzovus0ysfVm/JkToGL0REEjKyYuDkWTuxwVEkydiqYEaGlQ+3nHnkRQxeiIgkJLpi8Oj0EcjN9EvV2dWOxm2yrHzwzCNn8HgAIiIJRVr7a60sbH9oslQTpd3HBbDDrXfweAAiIpeTZWVBDyfKl7nykZwYvBARScptORV2HhfAwxCTG4MXIiKJuWllwa7yZa2tIgY23sfghYhIclaWOZvJjvJlrcMQv/3VQrz1YT1zYDyOwQsREQnRWtGwonw5+jVzv+THD9/ap5hTAwDPbKvr9D2e8uw9DF6IiEiTSFWP2UnGRs91isVTnr2HZxsREZEqPac3m3WWUiLnOsUjctZR9Ond1bUNaG3zVCcRT+HKCxERKTJS/pxokrHRc51EKCULs1+MuzB4ISIiRaLlzztrG5CS4usQrBhNMjZyrpOoeMnCWknAzJWRD4MXIiJSJFrWPG/tHpw5f7H980RWLaw6CTo/TrKwE431KHHMeSEiIkWiZc3RgQsQPx/G7NeMJhJXPDq9c7KwnsZ6JA8GL0REpChS/qx3zSGykrH47f26E1+1XtOHS6sov75nApbfVoJHp4+AyEv0zEjr9DW7GuuRuRi8EBGRokj5MwBDAYyRVQu114wuuZ44LBczSvojN9Mv9LzxAhA7GuuR+Ri8EBGRKqXy5x7dugo93siqhZ6S60QCENFVHj2N9ch6TNglIiJN8cqf28Jh3PHcnzUfa3TVQrTkOpHOvm48vZsYvBARkaDYM5Za28KmHweg9ZpK1yQSgLjt9G4CfOFw2FMtBEOhELKzsxEMBpGVleX0cIiIPCX2fKPTTS2Yt3YPgPhBg509UhJtNMfTqJ2lZ/5m8EJEREKUgoObR+fHPcn5tq8MxODc7rYGAgxA3IvBC4MXIiJTKXWhjYQFK28fg54Zfpw4ewGHTzbh1V1HEAg1t1/HVvukRc/8zWojIiJSpdWFFgCWbPgY4wtz4O+SgmWbDnYIXIDEmtYRxWLwQkREqoTPN/prg2aQY6RpHVEsBi9ERKRKtE9LdW0DW+2TLRi8EBGRKvE+LWIrKmy1T4li8EJERKpEu9B2SRGbUthqnxLF4IWIiFSJnDV08+h8LN98UPV52GqfzMLghYiINKmdNbTy9jF468N6zU2jMNhqn8xh2fEAP/7xj7FhwwbU1NQgLS0NZ86c0XzMnDlz8NJLL3X42pQpU1BZWWnRKImISJTSWUNa1UgRD5QNY58XMoVlwUtLSwu+8Y1voLS0FM8//7zw46ZOnYoXX3yx/XO/X+yocyIisl68s4ZEE3AH52ZYMSRKQpYFL4sXLwYArFmzRtfj/H4/8vLyLBgRERFZQTQBl4m6ZBbpcl7ee+899OnTB5dffjnuu+8+NDQ0qF7f3NyMUCjU4YOIiOwjWo3ERF0yi1TBy9SpU/Hyyy9j8+bNWLp0KbZu3Ypp06ahtbVV8TEVFRXIzs5u/ygoKLBxxEREJFKNxERdMpOu4OXhhx+Gz+dT/fjkk08MD+a2227DzTffjJEjR2LmzJl455138P777+O9995TfEx5eTmCwWD7x9GjRw2/PhERGaNWjfT0nVcyUZdMpSvn5cEHH8ScOXNUr7nssssSGU+n58rNzcWhQ4dw/fXXx73G7/czqZeISAJK1UhccSGz6Qpeevfujd69e1s1lk7+9re/oaGhAfn5jNiJiNwgXjUSkdksy3k5cuQIampqcOTIEbS2tqKmpgY1NTVobGxsv2b48OFYv349AKCxsRHf+973sHPnThw+fBibN2/GjBkzMHToUEyZMsWqYRIREZHLWFYq/dhjj3VoODdmzBgAwJYtW3DttdcCAA4cOIBgMAgASE1Nxf/+7//ipZdewpkzZ9CvXz987Wtfw5IlS7gtRERERO184XBY7BhQlwiFQsjOzkYwGERWVpbTwyEiIiIBeuZvqUqliYiIiLQweCEiIiJXYfBCRERErsLghYiIiFyFwQsRERG5CoMXIiIichUGL0REROQqDF6IiIjIVRi8EBERkasweCEiIiJXYfBCRERErsLghYiIiFyFwQsRERG5CoMXIiIicpUuTg+AiIjk09oWxq66Uzhx9gL6ZKZjfGEOUlN8Tg+LCACDFyIiilG5tx6L396P+uCF9q/lZ6dj0U1FmFqc7+DIiC7hthEREbWr3FuP+17Z0yFwAYBA8ALue2UPKvfWOzQyoi8weCEiIgCXtooWv70f4Tjfi3xt8dv70doW7woi+zB4ISIiAMCuulOdVlyihQHUBy9gV90p+wZFFAeDFyIiAgCcOKscuBi5jsgqDF6IiAgA0Ccz3dTriKzC4IWIiAAA4wtzkJ+dDqWCaB8uVR2NL8yxc1hEnTB4ISIiAEBqig+LbioCgE4BTOTzRTcVsd8LOY7BCxERtZtanI+n77wSedkdt4bystPx9J1Xss8LSYFN6oiIqIOpxfm4oSiPHXZJWgxeiIiok9QUH0qH9HJ6GERxcduIiIiIXIXBCxEREbkKgxciIiJyFQYvRERE5CoMXoiIiMhVGLwQERGRqzB4ISIiIldh8EJERESuwuCFiIiIXMVzHXbD4TAAIBQKOTwSIiIiEhWZtyPzuBrPBS9nz54FABQUFDg8EiIiItLr7NmzyM7OVr3GFxYJcVykra0Nn3/+OTIzM+HzmXOIWCgUQkFBAY4ePYqsrCxTntNL+P6o4/ujju+PNr5H6vj+qHPL+xMOh3H27Fn069cPKSnqWS2eW3lJSUnBgAEDLHnurKwsqf/hncb3Rx3fH3V8f7TxPVLH90edG94frRWXCCbsEhERkasweCEiIiJXYfAiwO/3Y9GiRfD7/U4PRUp8f9Tx/VHH90cb3yN1fH/UefH98VzCLhEREXkbV16IiIjIVRi8EBERkasweCEiIiJXYfBCRERErsLgRcPKlSsxePBgpKenY8KECdi1a5fTQ5LGtm3bcNNNN6Ffv37w+Xx44403nB6SVCoqKvCVr3wFmZmZ6NOnD2bOnIkDBw44PSxpPP300xg1alR746zS0lL84Q9/cHpY0nr88cfh8/lw//33Oz0UKfzwhz+Ez+fr8DF8+HCnhyWVY8eO4c4770SvXr3QrVs3jBw5Eh988IHTwzIFgxcVr732GhYuXIhFixZhz549GD16NKZMmYITJ044PTQpNDU1YfTo0Vi5cqXTQ5HS1q1bMW/ePOzcuRMbN27ExYsX8bWvfQ1NTU1OD00KAwYMwOOPP47du3fjgw8+wOTJkzFjxgzs27fP6aFJ5/3338czzzyDUaNGOT0UqVxxxRWor69v/9i+fbvTQ5LG6dOnMXHiRHTt2hV/+MMfsH//fvz85z9Hz549nR6aOcKkaPz48eF58+a1f97a2hru169fuKKiwsFRyQlAeP369U4PQ2onTpwIAwhv3brV6aFIq2fPnuHnnnvO6WFI5ezZs+Fhw4aFN27cGL7mmmvCCxYscHpIUli0aFF49OjRTg9DWg899FB40qRJTg/DMlx5UdDS0oLdu3ejrKys/WspKSkoKytDdXW1gyMjtwoGgwCAnJwch0cin9bWVqxbtw5NTU0oLS11ejhSmTdvHqZPn97hbxFdcvDgQfTr1w+XXXYZ7rjjDhw5csTpIUnjrbfewrhx4/CNb3wDffr0wZgxY7B69Wqnh2UaBi8KTp48idbWVvTt27fD1/v27YtAIODQqMit2tracP/992PixIkoLi52ejjS+Oijj/ClL30Jfr8f9957L9avX4+ioiKnhyWNdevWYc+ePaioqHB6KNKZMGEC1qxZg8rKSjz99NOoq6vDv/zLv+Ds2bNOD00Kf/3rX/H0009j2LBhePfdd3HffffhP//zP/HSSy85PTRTeO5UaSIZzZs3D3v37uWefIzLL78cNTU1CAaD+O///m/Mnj0bW7duZQAD4OjRo1iwYAE2btyI9PR0p4cjnWnTprX/96hRozBhwgQMGjQIr7/+Ou655x4HRyaHtrY2jBs3Dj/5yU8AAGPGjMHevXuxatUqzJ492+HRJY4rLwpyc3ORmpqK48ePd/j68ePHkZeX59CoyI3mz5+Pd955B1u2bMGAAQOcHo5U0tLSMHToUIwdOxYVFRUYPXo0li9f7vSwpLB7926cOHECV155Jbp06YIuXbpg69at+OUvf4kuXbqgtbXV6SFKpUePHvjyl7+MQ4cOOT0UKeTn53e6CRgxYoRnttYYvChIS0vD2LFjsXnz5vavtbW1YfPmzdyTJyHhcBjz58/H+vXrUVVVhcLCQqeHJL22tjY0Nzc7PQwpXH/99fjoo49QU1PT/jFu3DjccccdqKmpQWpqqtNDlEpjYyNqa2uRn5/v9FCkMHHixE6tGT799FMMGjTIoRGZi9tGKhYuXIjZs2dj3LhxGD9+PJYtW4ampibcfffdTg9NCo2NjR3ucurq6lBTU4OcnBwMHDjQwZHJYd68eVi7di3efPNNZGZmtudKZWdno1u3bg6Pznnl5eWYNm0aBg4ciLNnz2Lt2rV477338O677zo9NClkZmZ2yo/KyMhAr169mDcF4Lvf/S5uuukmDBo0CJ9//jkWLVqE1NRUzJo1y+mhSeGBBx7A1VdfjZ/85Cf493//d+zatQvPPvssnn32WaeHZg6ny51k99RTT4UHDhwYTktLC48fPz68c+dOp4ckjS1btoQBdPqYPXu200OTQrz3BkD4xRdfdHpoUvjmN78ZHjRoUDgtLS3cu3fv8PXXXx/+4x//6PSwpMZS6S/ceuut4fz8/HBaWlq4f//+4VtvvTV86NAhp4cllbfffjtcXFwc9vv94eHDh4efffZZp4dkGl84HA47FDcRERER6cacFyIiInIVBi9ERETkKgxeiIiIyFUYvBAREZGrMHghIiIiV2HwQkRERK7C4IWIiIhchcELERERuQqDFyIiInIVBi9ERETkKgxeiIiIyFUYvBAREZGr/H9o8LDb5LhVNQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x_samples, y_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Activation functions\n",
    "sigmoid = lambda x: 1 / (1 + np.exp(-x))\n",
    "identity = lambda x: x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialization of weights and biases\n",
    "weight_matrices = []\n",
    "bias_vectors = []\n",
    "activation_functions = []\n",
    "\n",
    "for (fan_in, fan_out) in zip(LAYERS[:-1], LAYERS[1:]):\n",
    "    kernel_matrix_uniform_limit = np.sqrt(6 / (fan_in + fan_out))\n",
    "\n",
    "    # Glorot uniform initialization\n",
    "    W = np.random.uniform(\n",
    "        low=-kernel_matrix_uniform_limit,\n",
    "        high=+kernel_matrix_uniform_limit,\n",
    "        size=(fan_in, fan_out),\n",
    "    )\n",
    "\n",
    "    # Zero bias initialization\n",
    "    b = np.zeros(fan_out)\n",
    "\n",
    "    weight_matrices.append(W)\n",
    "    bias_vectors.append(b)\n",
    "    activation_functions.append(sigmoid)\n",
    "\n",
    "activation_functions[-1] = identity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def network_forward(x, weights, biases, activations):\n",
    "    a = x\n",
    "\n",
    "    for W, b, f in zip(weights, biases, activations):\n",
    "        a = a @ W\n",
    "        a = a + b\n",
    "        a = f(a)\n",
    "    \n",
    "    return a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f1cde1be260>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAABSLUlEQVR4nO3de3wU1f038M9uIFdIQgjkAhEioBCBBMFggFbFKCAPQtunVdRHsJb+tPAUxfan9FEpP2wjta2g8BPFuxapvXiljQJBKRhEofEnoAgpt2ISJIFsCJBgdp4/1gmbzc7szOzMzpnJ5/168dJsZndPZi/nO+d8z/d4JEmSQEREROQQXrsbQERERKQHgxciIiJyFAYvRERE5CgMXoiIiMhRGLwQERGRozB4ISIiIkdh8EJERESOwuCFiIiIHKWb3Q0wm9/vx5dffomePXvC4/HY3RwiIiLSQJIkNDU1ITc3F16v+tiK64KXL7/8Enl5eXY3g4iIiAw4cuQI+vfvr3qM64KXnj17Agj88ampqTa3hoiIiLTw+XzIy8tr78fVuC54kaeKUlNTGbwQERE5jJaUDybsEhERkaMweCEiIiJHYfBCREREjsLghYiIiByFwQsRERE5CoMXIiIichQGL0REROQoDF6IiIjIUVxXpI66tja/hO0HGnCs6Sz69kxEcX4G4rzc44qIyE0sHXnZvHkzpk2bhtzcXHg8Hrz++uuqx7/33nvweDyd/tXW1lrZTHKJ8l01mLC0AjNXb8P8tVWYuXobJiytQPmuGrubRkREJrI0eGlubkZhYSFWrlyp63579+5FTU1N+7++ffta1EJyi/JdNbjz5Z2oaTzb4fbaxrO48+WdDGCIiFzE0mmjKVOmYMqUKbrv17dvX6Snp5vfIHKlNr+ExW/tgRTmdxIAD4DFb+3BNQXZnEIiInIBIRN2i4qKkJOTg2uuuQZbt25VPbalpQU+n6/DP+path9o6DTiEkwCUNN4FtsPNMSuUUREZBmhgpecnBysWrUKf/nLX/CXv/wFeXl5uPLKK7Fz507F+5SVlSEtLa39X15eXgxbTCI41qQcuBg5joiIxCbUaqOLL74YF198cfvP48aNQ3V1NR599FG89NJLYe+zcOFCLFiwoP1nn8/HAKaL6dsz0dTjiIhIbEIFL+EUFxdjy5Ytir9PSEhAQkJCDFtEoinOz0BOWiJqG8+GzXvxAMhOCyybJiIi5xNq2iicqqoq5OTk2N0MElic14NF0woABAKVYPLPi6YVMFmXiMglLB15OXXqFPbv39/+84EDB1BVVYWMjAxccMEFWLhwIY4ePYoXX3wRALBs2TLk5+fjkksuwdmzZ/H000+joqIC7777rpXNJBeYPDwHT9xyKRa/tadD8m52WiIWTSvA5OHWBcAsjEdEFFuWBi8ff/wxrrrqqvaf5dyUWbNm4fnnn0dNTQ0OHz7c/vvW1lbcc889OHr0KJKTkzFy5Ehs2LChw2MQyUKDhmsKsnFNQXZMA4nyXTWdAqacGARMRERdmUeSpHBpAo7l8/mQlpaGxsZGpKam2t0csogIQYNcGC/0AySHSk/ccikDGCIijfT038LnvBCFEqGabqTCeECgMF6b31XXBkREQmDwQo4iStDAwnhERPZh8EKOIkrQwMJ4RET2Eb7OC1EwUYIGrQXvMnskoLK6niuRiIhMxOCFHEWUarrF+RnITk1Ara8l7O89ANKTu+OeV6s6HMOVSERE0eO0ETmKXE1XaezCg0CAYHU13fV7anH2a79iGyQAJ06f6xTcxDKpmIjIrRi8kKOIUE1XXu108vS5sL9PS+6O9OTuYX/HlUhERNFj8EKOI1fTzU7rODWUnZZoeW0VtdVOMq/HoxjYAFyJREQULea8kCNNHp4T82q6QOTVTgDQ0Nyq6bG4EomIyBgGLySsSHsGxXk9KBnUO6ZtMjPgsDqpmIjIrRi8kJBEKP8fjtaAIyOlO040nws7veRBYIrL6qRiIiK3Ys4LCUeE8v9KtK52emj68PafQ38PWJ9UTETkZgxeSCiilP9XonW103Ujc21LKiYicjtOG5FQ9JT/j3W+i0xe7RQ6rZUdMq01eXgOJg7NwkuVB3Go4TQGZCTj/5QMRHw3XjMQEUWDwQsJRZTy/5FoWe0ULm/n6S0HbM/bISJyOgYvJBRRyv9robbaSc7bCZ3ckvN2OHVERGQcx69JKHJCrJpYlP+Phuh5O0RETsfghYQS5/Xg+kL1EYnrC3OEXqmjJ2+HiIj0Y/BCQmnzS3jzE/Wl0G9+UiP0qIVT8naIiJyKwQsJRUv5fdFHLZyUt0NE5EQMXkgobhi10FrITuS8HSIikTF4IaG4YdRCayE7kfN2iIhExuCFhOKWUQu5kB0r7BIRmY91Xkgo8qjFnS/vhAfosNzYaaMWWgrZERGRfh5JksRdtmGAz+dDWloaGhsbkZqaandzyCARd5Vu80sMRIiILKKn/+bICwlJtFELEYMpIqKuiiMvRBEolfqXwyjmsBARRU9P/82EXSIVLPVPRCQeBi9EKqIp9d/ml1BZXY83qo6isrqeAQ4RkUmY80KkwmjRPObIEBFZhyMvFFNOG43I7JGg+zg5RyZ0xKa28SzufHknynep791ERETqOPJCMePI0QitsdU3x0XKkfEgkCNzTUE2l1kTERnEkReKCaeORhxvbtF1XDQ5MkREpA2DF4papKkgJ6/Y0bvXkhs2liQiEh2njSgqWqaC9IxGlAzqbXWTdZH3WqptPBs2+PIgsF+RvNdStBtLsoovEVFkDF7IMKXibfJUkFy8zcmjEXr3WtIb7ARzZE4QEZENOG1EhuiZCop2NMJuenaIloMdAJ12xlbbWNKpOUFERHbgyAsZomcqKJrRCFHo2WtJDnZCR1GyFUZRuEKJiEgfBi82c2qOg56pIL1TLzLRzk2c16M5J0dPsOPknCAiIjsweLGRk3Mc9E4F6R2NcPK5kWkNdrQGgrW+s6isrhcmmCMisoulu0pv3rwZjzzyCHbs2IGamhq89tprmDFjhup93nvvPSxYsAC7d+9GXl4e7r//fsyePVvzczplV2mn71Tc5pcwYWlFxKmgLfdO7NDBahlNcfq50auyuh4zV2+LeFxGSnc0NJ9r/9lpwRwRkRphdpVubm5GYWEhVq5cqen4AwcOYOrUqbjqqqtQVVWFu+66Cz/60Y/wzjvvWNnMmHNy3ROZ0cRUeTRielE/lAzqHXaqyCnnxqytDuScoEhjKMGBC8BkXiLquiydNpoyZQqmTJmi+fhVq1YhPz8fv/vd7wAAw4YNw5YtW/Doo49i0qRJVjUz5tyS46B3KkgLp5wbM6e1IuUEKYVETOYloq5KqJyXyspKlJaWdrht0qRJuOuuu+xpkEWcXPcklJ7EVC2ccG601rfRQykQzEiJR31zq+L9RAnmiIhiSajgpba2FllZWR1uy8rKgs/nw5kzZ5CUlNTpPi0tLWhpOb//jM/ns7yd0RK97oneVT56VuFE4oRzY9Wy5nCBYG3jGdz96icR7+uEQJeIyCxCBS9GlJWVYfHixXY3QxeR654YmQ4xc0mzyOcGsH5aKzQQrKyu13Q/UQv8ERFZQagKu9nZ2airq+twW11dHVJTU8OOugDAwoUL0djY2P7vyJEjsWhqVIwmu1rNSJXX8l01mLC0AjNXb8P8tVWYuXobJiytMJxEKuq5kcV6WitSMq8HgeBS5AJ/RERmEyp4KSkpwcaNGzvctn79epSUlCjeJyEhAampqR3+OYGekvOxYGSVj1Ul7UU7N8FiPa0lejBHRGQHS6eNTp06hf3797f/fODAAVRVVSEjIwMXXHABFi5ciKNHj+LFF18EANxxxx1YsWIF/vM//xM//OEPUVFRgVdffRXr1q2zspm2MTvZNRrb/lWvazrE6pL2Ip2bYHZMaykl8/ZK6Y7vFPVDWlI82vyS7no6REROZWnw8vHHH+Oqq65q/3nBggUAgFmzZuH5559HTU0NDh8+3P77/Px8rFu3DnfffTeWL1+O/v374+mnn3bVMulQZia76hHcuR083oxntx7QdD95OiQWS5rtOjdqjG51EK3gYG79nlq8XvUlGppb8czWg3hm68EOeUluqE5MRKTG0gq7dnBKhV07hevctHplzuUoGdQbb1Qdxfy1VRGPX35jEaYX9TPQSrHZFSBEqj7842/n46nNB7pMdWIicg89/bfjVxuRPkqdXySh0yGiL2m2mh3TWlqm6lb/o3PgEvx7FrQjIjdg8NKFqHV+WgRPh4i+pBnQlvcRTW5IrKe1tEzVqY2jsqAdEbkFg5cuJFLnpyQ9uTse/u6IDtMNduV+aKVlWsdpuSFmLb9mQTsicjqhlkqTtYx2Witnhs+TEHVJs9oS7jte3on/ems3lm/YZ8kybyuZNQXn1qk8Iuo6OPLShejttOSpn8tVphhEW9KspV7Ns1sPKt5f5NwQLVN1Hg+gtLm1CFN5RERm4MhLFxKpWmswPVM/cu7H9KJ+KBnU29YO3+jUWLDg3BAj2vwSKqvr8UbVUVRW13co7BcNLQXr5nwrPxDEKPyeBe2IyA048tKFqOWphMrWmPshWjE0M/M5jDyW1Xk0SgXrgl+vURf0Uv09EZHTsc5LF6TUwd542QUYmJmsOQgRMeG1sroeM1dvM+Wx5Jo2WkWqwWJmHlCkoFG0oJKIKBI9/TeDly6q9Ws/Xqo8iEMNpzEgIxn/p2Qg4rtpn0WMZUetR5tfwoSlFYp5IVrIuSFb7p0IAJqCAPl5laasgh+TQQQRUWcsUkeqwo2YPL3lgOYRE6v3NYqGnqmxcIJzQ9bvqdU8shSL7RKIiCiACbtdjBk7QevpqO2gtIRbC3mZNwBd50lrfoxTa6xYlYRMRGQER166ELNGTLR2wFv3f2VbzkXwEu4Ne2rxjMry6LtLh2BgZkp7OwFgwtIKXefJzdsliJjbRERdG4MXgVidZGnW1IbWDnjFpur2/7ejs5OXcDeeaVUNXi7O7tmhXZXV9brPkxO2SzBCKbdJHoHiRo9EZAdOGwmifFcNJiytwMzV2zB/bRVmrt6GCUsrTK30atbUhp56MTK7KtfKo01K5FGU4GkQI+dJSw0Wp9VY0VLwL/TcERHFAoMXAZiRhxJJm1/C8aYWTcf27ZmomuOg1lErsauzM5KfY3QKSNTtEowSPbeJiLouThvZLBYrd8LlLIQjT22caG7ptOw3dNpHqViaGrmze3T9Xowf3CcmeTBGRlGimQISbbuEaLg9CZmInIvBi82sXmKrlLMQSu5ary/Mwdw1/9SU4xDaUe+rO4UVm/ZHbNOKTdVYsam6Q0BkVb6PkVGUaHfMlnNtnE7ruTt4/LTFLSEi6ojTRjaz8upWbVQnVHZaIlbedCne/KRGV45D8L5G4wdn6mqfHBCV/W2PZfk+kfJzPAiMKoWOorhtCsiI4vwMZKcmRDxu7UeHmfdCRDHFkRebmb3ENngE43hTi6YpnQemDsPs8flRjwJFmm4J93gA8OTmA51+Z9ZqlmhGUdw0BWREnNeDmcUX4NEN+1SPY/E9Ioo1Bi8205JfkZESj9rGM6isrlftPLXmtoTK7JmAOK8n6lGgaKvbBjOzUq+WzQyVuGUKyKiBmSmajmPeCxHFEoMXm0Xq8CUA9c2tuPvVTwAo10vRmtsSjjyqY8YokJFEXiVmltTv6qMoRrm5+B4RORdzXgSgp5x9uOXTenJbgoXmexjNDwk1eXgOttw7Ea/MuRzzrhqss1WdmXVVH5yfUzKoNwMXDcx6TxARmYnBiyCCO/xHbyhCRkr3sMeFS5yNlKsSTrh8DzMLrcmBwt3XXKS7oF0oXtXbx43F94jI+Ri8CETu8LNTE9HQfE7xuNDiYEZGJpRWzZi9ysZIQTtZLK7queFgZErviYyUeNw2fiDSkuJ53ogoppjzIiC9ibNaRyYemDoMmT0TOuR7hKuvYnZ+iFIeTE5aIq4vzMFTmw90mvKKxVU9NxzULnSjy9eqjqK+uRXPbj2IZ7ce5Hkjophi8CIgvUmSWivCzh6f3yEQiNR5m7nKRikgWr+nFmnJ/8bJ0x1HmtKTu6PsuyNM6QzDBWjr99Ryw0Gd4rweNJ4JBCw8b0RkJwYvAtJbnt5ILRM7dgsOXXastkLqxGnlaTM9wgVo2amJOPt1m6VbMrhRLLayICLSgjkvAjKSJKknV0WE3YIjrZAKt9tz8H215KkobnjpO9tppCeYUzYcjHW+DjdqJCJRcORFUFoKq4VOh1xTkK0pV8Xq/ZS0MNoGrXkqRpePBxO58Jod+TrcqJGIRMHgRWBqibPRdF4idEJG2qBnqsvI8vFQarlHVm0kqYUdU34AC9YRkTgYvAguXHn6aDsvETohvW3Qm28RTeAVmlMUys5VSnbmnYwe0AteD6A2O+X1BI4jIrISc14cJlLnJQG47y+fYuv+44o5EHqrplqRW6G3DXrzLYwGXpGWaCvm0YSpfGwFO/NOdhw6oRq4AIHAZsehE6Y/NxFRMAYvDqNlOuTkmXO4+ekPMWFpRdjOVE9CcPmuGkxYWoGZq7dh/toqzFy9TfFx9dCblKx3mklLcNQruTuyUxM63J6W1B13lQ7BNQXZne4jQqKznVN+Ikw3EhEBDF4cR0/HUNN4Fne8vBPLN+zr0KG2+SWkJQWqo/ZKie9wn+DVSVaPMuhZIaV3mklLcFT23RHYet/VuLv0IqQnBbZjOHnmHB7dsC9sgCbCahs7p/xEmG4kIgKY8+I4RjqGRzd8gVe2H8Ivr78EADrla2SkdMd3ivqhtCC7Q+XdWORWaK3mq7f2jfzYkVZsle+qwbINX2jKHxJh5MHIeXDDcxMRBWPw4jCROhAltb4W3PHyzrC/O9F8Ds9uPYjLgoKGWC6nDpeUHO4YvYX4APXgSG+AJsLIg9Hz4PTnJiIKxmkjh4lmo0Ml4fI1RBhlCGV000g5OJpe1A8lg3obCtAA/UnGVjF780ynPDcRkYwjLw6kNB0SjdCRFBFGGcIxc9NIvQGaSCMPZm+e6ZTnJiICGLwISUsBNLkD2VZdj7lrduLkGXP2AgpdrSNifoOWaSYtjARoWvJoYsWs8+C05yYiYvAiGD0F0OK8HowfkomHvzdCMZ9Fr9DVOiKMMljFaIDGkQciInsx50UgRpcmTx6eg9vHD4zqucPla7g9v8HIBpjB9w2XR2OlWG/ESEQkqpiMvKxcuRKPPPIIamtrUVhYiMcffxzFxcVhj33++edx2223dbgtISEBZ8+6u/BVtEuTSwuy8czWg5qeS20kBQAqq+t1b/boVCJNA6mxc0sCIiLRWB68/PGPf8SCBQuwatUqjB07FsuWLcOkSZOwd+9e9O3bN+x9UlNTsXfv3vafPR53dJRqol2arGUJda/k7vjVjOFYsu6zsB01AExYWtHlOkjRp4HM3IjRyg0l7dyskoi6FsuDl9///veYM2dO+2jKqlWrsG7dOjz77LO47777wt7H4/EgO7tzeXY3i3ZpslqOChAYXSn77ghMHp6DScNzOnUy6/fU2rJTsShETUA1s1iglaM3HBkioliyNOeltbUVO3bsQGlp6fkn9HpRWlqKyspKxfudOnUKAwYMQF5eHqZPn47du3crHtvS0gKfz9fhnxNFuzRZreR/TkiOSmi+BgDb9+yh8MzaksDKrR7s3qySiLoeS0dejh8/jra2NmRlZXW4PSsrC59//nnY+1x88cV49tlnMXLkSDQ2NuK3v/0txo0bh927d6N///6dji8rK8PixYstaX8sRbM0OdxVb7iS/0piWU2X9DGjWKCVWz3EahsJIqJgwq02Kikpwa233oqioiJcccUV+Otf/4o+ffrgySefDHv8woUL0djY2P7vyJEjMW6xOSJVzpUAPDB1WKcOQOmqVy7533imNWKnIWI1XQowo1iglRtKirBZJRF1PZYGL5mZmYiLi0NdXV2H2+vq6jTntHTv3h2jRo3C/v37w/4+ISEBqampHf45ldLSZNmSdZ91GIKPdNULaJvuEbWaLpmzJYHWoHPr/uO6pwYZ+BKRHSwNXuLj4zF69Ghs3Lix/Ta/34+NGzeipKRE02O0tbXh008/RU5O10j6mzw8Bw9MLQj7u9AcArOuekXZs4c6i6YWjUxr0Lli035MWFrRKUBWqy3DwJeI7GD5aqMFCxZg1qxZGDNmDIqLi7Fs2TI0Nze3rz669dZb0a9fP5SVlQEA/uu//guXX345Bg8ejJMnT+KRRx7BoUOH8KMf/cjqpgqhzS9hybo9YX8XmkNg1lVvV6im62TR1qLRsxN58OoyABFXEIm8jQQRuZflwcsNN9yAr776Cg8++CBqa2tRVFSE8vLy9iTew4cPw+s9PwB04sQJzJkzB7W1tejVqxdGjx6NDz74AAUF4Ucj3EbPaIqZV71aO0gRanmI0IZYi6YWTaRl9MHkAHnhXz/FidOd98sKXTrPwJeI7OCRJMlV6199Ph/S0tLQ2NgYk/wXszvSN6qOYv7aqojHLb+xCP9rZC4mLK2IeNW75d6Jmtuk9veIUMtDhDY4VbhzZ0S49xVfFyKKlp7+m8FLFMIvUY7HQ9OH47qRxr6wK6vrMXP1tojHvTLncpQM6t2+2ggIf9VrVnE5pSqvZj+P6G1wuja/hEfX78WKTdVRP5b8Hgx+7K42IkZE5tHTfwu3VNoplJYoNzS34idrduJX63Yb2kRPb/JsLDZPNGtVk9Pb4AZxXg/GD+5jymOF5lLZsVklEXVNMdmY0W3UOlLZ6n8cxOp/HGz/WesQupEcAqv35hGhiJ0IbXALPQm8ariCiIjswpEXAyJ1pOHoKZVuZDTFyqteEWp5iNAGt4i0/NoDID25O5fOE5GwOPJigJEOUm+pdJF2OhahlocIbXCTSKvLAHAFEREJi8GLAUY7SL1TG6LsdCxCLQ8R2uA2kQLkaGrLEBFZicGLAcX5GchIiUdDc6uh+zttakNLHs4DU4dZOkrEeiLWUAuQRRr9IyIKxqXSBv3tf2rwkzU7Dd03dImpUyjV8ri+MAdvflITkxofrCdCROROrPMSozovZX/bgyc3H9B8vJGicaIJreVxorkVc9fEtvYK64nEDs81EcWKnv6b00ZRWHhdAQr7p+P+N3ahoblzKfVgbpnaCJ5maPNLmLC0QrH2ip4EZaNtIOtwlIuIRMXgJUrXjczFpOE5nUYjlqxzf6Ija6+4l1I149C9jYiI7MDgxQThRgImDXd/oiNrr7hTpGrGVo2oERFpxeDFIl1haoO1V+xhdR6K1hG1R9d/gfGDM10ZmBOR2Bi8kGGsvRJ7schD0TpStmLTfqzYtF/X8zMBmIjMwO0ByLBIZeYB5ycoi0RpM1A9W09ooXekTOvzl++qwYSlFZi5ehvmr63CzNXbMGFphWntJqKug8GLRm1+ydAu0W4Xi12tKba7akfa2dzI88cq8LIavweIxMBpIw24ZFQdK7FaL5Yru9SqGRt5frckAPN7gEgcHHmJwC1XjFazcldriv3KLqURNSPPryfwEhW/B4jEwuBFRSyH6onU2LGya/LwHGy5dyJemXM5fnLlhZruk5mS0Ok2py+p5/cAkXgYvKhwwxUjuUOkPBQPAlMYZq/skkfUxg/uo+0Ons55IeECmnBEXVLP7wEi8TDnRYXTrxjJPezeVfv4qRZNx1V8Voef/emTjtWlUxORntwdjafPOXJJPb8HiMTDkRcVLMJGIrFzZZfW9/gzWw92GqWo853FyW8CFycuqef3AJF4OPKigkXYSDR2rezS8lnweIBwaR9y0JKe3B0J3byo9Z0fxXHCnl/8HiASD4MXFbEcqmflUdLKjq0nIn0WJACSSr6qBODE6XP4w4/GwuvxOOp9bveUHRF1xuAlAnmoPrS+Q6+U7vhOUT+kJcWjzS9F9cXF+hHkBEqfhbTk7ii5sDf+vqs24mMcP9WC6UX9rGymJZT+dieMHBG5kUeS1K6XnMfn8yEtLQ2NjY1ITU017XHlkZH1e2rxetWXaGhubf9dNIGGXD8i9EWQQyFWqSXRtPklrKjYh+e2HsTJM+d03feVOZc7esNSjpASWUdP/82EXY3ivB40nmnFc1sPdghcAOOFqlg/gpxo/Z5aLNuwT1fgYtVS7lhjMUYiMTB40ciKQIP1I8hp1D4HSpgXQkRmY/CikRWBButHkNNE+hyEw006ichsTNjVyIpAg/UjyGm0vr/nXTUIQ7J6om/PRIwe0As7Dp3AG1VHmSdCRKZg8KJRtIFGuEQ/1o8gp9H6ORg/uA9KBvVG+a4aXPHIJq6kIyJTMXjRKJpAQ20pNOtHkJPo+RworaSTE9w5lURERjHnRSO5UBWgr8S5/AUemicgf4EDsK3kO5FeWj8HALiSjogswzovOukpKNfmlzBhaYVigqN8lbrl3okAwPoR5BiRPgeV1fWYuXpbxMdxet0XIjKPnv6b00Y66dlbRs8KpZJBvfklTo4R6XPAlXREZCUGLwZo3VuGX+DkZmqfA7tW0llVAZeVdYnEwuDFQlwKTV3VieZWeBV2mQasWUlndI+wSIEJ9x4jEg+DFxMofflxKTQ5jRkjDOW7ajB3TedVRqHMXElndGVTpMCEK6aIxMTgJUqRvvy4FJqcwowRBi3bB3g9wIqZo0zr9CNt3eFBYGXTNQXZnUZU1AKTlTeNwpJ1n+l+XCKyHpdKRyHSMujyXTWYPDyHS6FJeFrey1po2T7ALwG9UhIMt1Xvc4bbukPLXmX3v7GLe49Rl9bml1BZXY83qo6isrpeqNIGHHkxSM/Vnp4VSkSxZnTkIhw7ktSNPKeWgKehWduu2Uy4JzcSPdcrJiMvK1euxMCBA5GYmIixY8di+/btqsf/6U9/wtChQ5GYmIgRI0bgb3/7WyyaqYveqz15Zcb0on4oGdSbgQsJw8xNR+1IUjfynGYGHEy4J7cxayTWSpYHL3/84x+xYMECLFq0CDt37kRhYSEmTZqEY8eOhT3+gw8+wMyZM3H77bfjn//8J2bMmIEZM2Zg165dVjdVFy6DJrcw870sJ6krheYeBK7ezExSN/KcWgOOjJT4mP4tRHbTMqUqQnVsy4OX3//+95gzZw5uu+02FBQUYNWqVUhOTsazzz4b9vjly5dj8uTJ+PnPf45hw4ZhyZIluPTSS7FixQqrm6oLl0GTW5j5Xja6jUY0jDyn1oDnoenDdT0ukdOZORJrJUuDl9bWVuzYsQOlpaXnn9DrRWlpKSorK8Pep7KyssPxADBp0iTF41taWuDz+Tr8iwU7rjCJrGD2e9mqJHW15EG9z6k14LluJBPuqWtxyqyCpQm7x48fR1tbG7KysjrcnpWVhc8//zzsfWpra8MeX1tbG/b4srIyLF682JwG6yB/+XEZNDmdFe9ls5PUtSQP6n1OOeAJfdzsKB+XSFRa6jg5ZVbB8auNFi5ciAULFrT/7PP5kJeXF5Pn1vrlRyQ6K97LWrfRiERPoTi9z6klMOHWAOQGWlcPOaW4qqXBS2ZmJuLi4lBXV9fh9rq6OmRnZ4e9T3Z2tq7jExISkJBgXs0IvXhVRm4h4nvZzGXcStQCHtGXixJpofcCwAmzCpbmvMTHx2P06NHYuHFj+21+vx8bN25ESUlJ2PuUlJR0OB4A1q9fr3i8CLgMmtzCyveykYJXdiYPOmG5KHVdWj9PRlYPOaG4quXTRgsWLMCsWbMwZswYFBcXY9myZWhubsZtt90GALj11lvRr18/lJWVAQDmz5+PK664Ar/73e8wdepUrF27Fh9//DGeeuopq5tqCIeUiSIzOoJhV/JgLEZ8iIzS83nScwEQPAIp4khsMMuDlxtuuAFfffUVHnzwQdTW1qKoqAjl5eXtSbmHDx+G13t+AGjcuHFYs2YN7r//fvziF7/AkCFD8Prrr2P48OFWN1U3DikTRRbN5oZ2JQ8a/cInkll1Yav386Q1sK/1nUVldX2n9or6/vZIkiTOZgUm8Pl8SEtLQ2NjI1JTUy17HqU3kPzWFGVojUgPs79w2/wSJiytUAwE5OS/LfdODPs88v0jJQ8q3d+oN6qOYv7aqojHLb+xCNOL+pn2vOQOVl3YGvk8VVbXY+bqbREfOyOle4ctMey4ENfTf3NjRgOcUoGQSI/yXTWYsLQCM1dvw/y1VZi5ehsmLK2IKrcj2pwVO4reAc5ZLkrisTJXysjnKVIdJ1noXl6i53YxeDHAKRUIibSy6gvXjJwVO5IHWYSSjLD6wtbI50nLBUA4ol+IO77Oix2cUoGQSAsrk1PNGsGIdfKgU5aLkliszpUy+nlSquOUkRKP+uZWy9prJQYvBnBImdzEyi9cMwtexTp5kEUoSS+rL2yj+TyFuwCobTyDu1/9RFd7RVlhy+DFAKdUICTSwsovXKePYIi+XBQQpzMh6y9so/08hV4AVFbX62qvSCtsGbwY4PQvZKJgVn/hOn0EQ6TloqGByonmFixZ95kQnQnF5sLWzM+TnvZGU/LAClwqHQWRolAio2K1HDkWIwRuHoUI930TDss12Evu5IHwF7ZmvS5mvde1tPeaguyoSh5opaf/ZvASJTd/WVLXEasvXCuF69wzUuLx0PThuG6k2G2PROmqV4lV9W9IG7subI32R5Haq7VWzCtzLo9qlFJP/81poyiJNKRMZJTTp3aUOveG5lb8ZM1O/Me/87HwugJb2hYttdVgSkReJdIV2JErFU3AFKm9Iq6wZfBCRACckZwajpbO/cnNB1DYPx3XjcyNWbvMEmk1mBqWa7BPLC9szchHUWuviCtsWaSOiNo5cYd0rZ37/W/sErLYViTRBCAs1+B+saj4LmLRRgYvRF1Im19CZXU93qg6isrqekd25qG0du4NzeccWfXaSADCCsBdRywqvtu1TYcaThsRdRFuXR2np3N34jSKfNWrdeqI5Rq6Fq3v6a37v4pqGli0vDgGL0RdgGg1GsxUnJ+BjJR4NKiUOZc5cRolzuvBA1ML8JM1OzUd75QkazKH1vf0ik3V+MvOo1G9N0TKi+O0EZHLuX0X9DivBw9NHx7xOCdPo/RKidd03ANTh2HLvRMZuHQhxfkZSE/urulYM3aKFiUvjsELkct1hV3QrxuZg//4dr7i7z1w9jSK1qmBzJ4Jjv0byXpuuFiRMXiJkhsTIMldRKzRYIWF1xXgv28ahYyUjlehOWmJjp4WA8Rcqkpi2H6gASdPn9N8vBsuVgDmvETFrQmQ5C5dqeO7bmQuJg3PEWJO3kzcDJaUGL3ocPrFCkdeDJITIEOH482YUyQyk4g1Gqwkypy8mURcqkpiMHrR4fSLFQYvBrg9AZLchR2fO8hLVbPTOnY62SZOi3Ea3HkiXZyEcsvFCqeNDNCTAMl9RUgEotVo6IrM2MTVyqWqnAZ3Jvni5M6Xd8IDqG6T4aaLFQYvBnSVBEhyF5FqNHQ1ZgYGVuyZ4+Y6QF2B0sWJ1wMED56pXayYEVzHEoMXA7pSAiS5C3dBjz3RA4NI0+AeBKbBrynIFroz6+rCXZyMHtALOw6diBiQOHHUjcGLAcz8JyItnBAYcBrcPcJdnER6zUQPrpUwYdcAJkASkRZOKBDIafCuy8mLTxi8GBSLzH8icjYnBAacBncOs1eDOSG4VsJpoygwAZLIGk5IHtTSRicEBpwGdwYr8lKcEFwrYfASJSZAEpnLCcmDWtvohMBAbaktp8HFYFVeihOCayWcNiIiYTihcrWeNjolP47T4OKyMi/lRHNrxGN6JXcXctSNIy9EJAQnrMwx0kanFAjkNLiYrFoN1uaXsGTdnojHnTh9Duv31ArzPpUxeCEiIcR6ya6RvBqjbXRKYMBpcPFYlZcS6b0czO6LhnAYvBCREGKZPGg0ryaaNjIwICOsykvR8zkSsc4Pc16ISAhWJw/Ky0yXvLUbdxjMq3FygiM5k1W7wlsZ7MQCgxciEoJVX9JAYKRlwtIKzFy9Dc9sPRj2GLXkRznwqW08g4yUeEvaSBSOVUnf8udNK9ECcgYvRCQEq76klVYHhROuKFdw4HP3q5+goblVcdmz0TYSqbFiNZj8eYv0ThU1IGfOCxEJw+yVOWqrg9TIQ+RK9TXCEW31ELmLFUnf8uftvr9+ipOnz3X6vcgBOYMXIhKKmV/SelZUBOvbM1FT4JOSEIeHZoxAdqqYq4ei4YQqx12NFUnf8udtRcU+PLf1IE6eOR/EiByQM3ghIuGY9SWtN8kwuOKtlsCnuaUNh+tP4/rCXFd19E6ockzmifN6ML/0IsybOMQx72MGL0TkWnqSDEOHyLUGPk9trsYr2w+h1tfSfpuTO3qrStGT+Jy0nJ8Ju0TkWpFWMAULTX7UGvg0t7Z1CFwAsbYz0MPKUvROZPYuzmQeS4OXhoYG3HzzzUhNTUV6ejpuv/12nDp1SvU+V155JTweT4d/d9xxh5XNJCKX0rKC6YfjB+KVOZdjy70TO22qmJ7U3dDzOrWj11NB2O2CV5nNX1uFmau3YcLSCscFpG5lafBy8803Y/fu3Vi/fj3efvttbN68GT/+8Y8j3m/OnDmoqalp//eb3/zGymYSkYupLTNddculeHDaJSgZ1LvT3H6c14Pbxg80/LxO7OhjWeVYZFZtEMqRHPNYlvPy2Wefoby8HB999BHGjBkDAHj88cdx3XXX4be//S1yc3MV75ucnIzs7GyrmkZEXYzRFUzzJg7Bcx8cDLuMVCsndfSsIGxs800tK7OYBG0uy0ZeKisrkZ6e3h64AEBpaSm8Xi8+/PBD1fv+4Q9/QGZmJoYPH46FCxfi9OnTVjWTiLoIORlxelG/sCMtSvd5+LsjNOXMKHFSR29llWOn0Dt1pmV6yaqRnK7MsuCltrYWffv27XBbt27dkJGRgdraWsX73XTTTXj55ZexadMmLFy4EC+99BJuueUWxeNbWlrg8/k6/CMiMos87RRaSj0nLRHpyd2F7OiNTk9YVeXYSfRMnWkJSpgEbQ3d00b33Xcfli5dqnrMZ599ZrhBwTkxI0aMQE5ODq6++mpUV1dj0KBBnY4vKyvD4sWLDT8fEVEkwdNOtb6zaDjVgoyUeBxuOINlG76AB+jQOent6M0sCBft9ITZVY6dRutIWWaPBPzsT59EnF7qmdhd80iOU5Ypi0B38HLPPfdg9uzZqsdceOGFyM7OxrFjxzrc/vXXX6OhoUFXPsvYsWMBAPv37w8bvCxcuBALFixo/9nn8yEvL0/z4xMRaRHn9aDxTCt+U/55h84oPTmwIik4L0ZPR29mLoRZNVqsKEXvFPLUWW3jWcU9rLLTEgEJmoKSyup6Tc/rpNwoEegOXvr06YM+ffpEPK6kpAQnT57Ejh07MHr0aABARUUF/H5/e0CiRVVVFQAgJyf8By4hIQEJCQmaH4+IyAilwKDx9DlIAO4uHYKBmSm6OnozC8IZSTRV46SCZWaSp87ufHmn6oja8eaWMPcOR9t0kJNyo0RgWc7LsGHDMHnyZMyZMwfbt2/H1q1bMW/ePNx4443tK42OHj2KoUOHYvv27QCA6upqLFmyBDt27MDBgwfx5ptv4tZbb8W3v/1tjBw50qqmEhGp0hIYrP3oCP7XyFzNycBm5EIE57Y8v/UAa7SYRMsuzlqDjZILMyMWSvR6gBOagyECLN4e4A9/+APmzZuHq6++Gl6vF9/73vfw2GOPtf/+3Llz2Lt3b/tqovj4eGzYsAHLli1Dc3Mz8vLy8L3vfQ/333+/lc0kIlKlZwWK1tGKaB8z3HSTFpye0CbS1JnW6aXLB/VuH8lR4peAuWv+iSe8Hstzityy4aalwUtGRgbWrFmj+PuBAwdCks6/7Hl5eXj//fetbBIRkS5tfglb9x/XdKyewGDDHuVVl5EeU2m6SQtOT2inNnWmdXop7puAZOVNl2LeKzuhtqhIz7SeEW6qNcO9jYiIFMg1PFZs2q/peK2BQfmuGjyz9aChx1SbblLTFWq0xJqW6SXZvmNNqoGL1dN6bqs1w12liYjC0Du6oTUwkIOPSORph9DHjDTdpPRYgPtrtNhBy8qs8l01eHTDPk2PZ8W0ntnJ3CJg8EJEFMLI6Mb1hTmavvi1Bh8SwgcbRjq3rlKjxS5q00tag1WZFdN6VuRs2Y3BCxG5ltHkRCOjG29+UoP/nDws4uNrDT5+OH5g2GBDa+f2wNRhyOyZ4OikTDfQ816yalrPjRtuMnghIleKJjnRyJe41itXrcHHNQXhi3lqXeUye3w+AxYB6HkvWTWt58YNN5mwS0SuE21yotEvcS0d1egBvZCR0l3x95ESa7n/kLNofS/dXXqRZdN6btxwk8ELEbmKGcXfivMz0CtZ/8B0pI6qfFcNrnhkExqaz4X9vdbgQ88qF1JmdANLPSIFDgCQnZqAeRMHm/7cMjcGvJw2IiJXMSM5Mc7rwaySgVi2UdsSaSCwx5HalauW1Ut6Emu78v5DZohVzRMt9WB+ef0llr9ubttwk8ELEbmKWcmJ+X166Hrek6fP4Z1dteiVEt8pmGj92o9fvPapauDSOyUe7//8KsR30z4g3lX3H4qWmXtKaRFN4GBmRVw3BbwMXojIVcxKTjSS9xJaQTUnLRHXF+bgTzuOKk4VyeqbW7Hj0AkGIxazq+aJkcDBitEhtwS8zHkhIlcxKzlRS65CqNCUiZrGs3hy8wE0NLdqur+Tlqo6lZ5pRbPJgcP0on4RN/B0W0VcszF4ISJXMSs5Ue1xrHLw+OkYPVPX5YSaJ2Yknbsdgxcich2zVuMoPY5Vlm34ostfUVvNCTVP7BwdcgrmvBCRK5mVnCg/zrbqesxdsxMnz6jnrkTLjj1mzEwKjSUj7dZa5M/OmidOGB2yG4MXInIts5IT47wejB+SiYe/NwJ3vrwTAHTv6qyFHXvMxGrJsNmMtlvL0mW7a544YXTIbpw2IiLSKFbTSEauqI0UXHNqUmi07Ra9yJ8bK+KajSMvREQ6hE5HHW9qwZJ1n5n6HHqvqI2MQti1ZDhaZrVb5JonThgdshtHXoiIdApe8prZM8G0xzVyRW10FMJIUmgsyulHYmYyq56ly0qsOieijw7ZjSMvRERR0DpK8sDUYcjsmYCDx09j2YYvAER/RR3NKITepFBRcmNESma1+pyIPDpkNwYvRERR0Lp6Zfb4/PZOZ0jfFNz/xq4OVXeN7DETzT5OepJCY11OP1J7zDxOjdpqplidk9Ckc3mkp6sHMwxeiIiioDc/oXxXDZas+6xD4JKREo8Hpuq/Wo9mFEJr0DV6QC9c8cgmYXJj9C51NroMXG1U5ZqCbFvyhUQZ/RIBc16IiKKkNT9BKT/lRHMr5q7Rv7onmlEIrZWIdxw6IVTBND0VlMt31WDC0grMXL0N89dWYebqbZiwtCLieY6UR7SiYl/Mz4lTV4ZZhcELEZEJJg/PwZZ7J+KVOZdj+Y1FeGXO5dhy78T2wMWKku/RLqnVEnSJlGMi09Juo529ltfpua0HNbXTrHPC7QI647QREZFJ1IriRZOfovZ80S6pjZQUKmrBNLV2R5PIrOV10lpl2axzYsV7x+kYvBARxYBVIxjyKERoLoSeBGC1oEtLjklGSjxqG8+gsro+pgmkSu2OprPXev7Tk7qj8cy5mGwxIOLol90YvBARxYCVIxhWLqlVG93BNz/XN7fi7lc/ASBGAmk0nb3W83/b+Hws2/BFTIrIiTr6ZSfmvBARxYAVJd+DC6RtP9CA4vyMqAquKdGzLYIICaTRdPZaX6d5EwfHrIgctwvojCMvREQxYHbJ91gvmw0e3an1ncWSt3d3WO4tE2FrgWh2jtbzOkUa8TJrt25uF9CZR5IkV6Un+3w+pKWlobGxEampqXY3h4ioAzOCDqUCaXLXZXXRuMrqesxcvS3ica/Mudy2BFL5HAHhO/tI5yja18mK4NLoHlZOqdCrp/9m8EJEFGPRdChtfgkTllYoJqTKowpb7p1oWSf1RtVRzF9bFfG45TcWYXpRP0vaoEW0AUQ0Be6sCi71tMlpRe309N+cNiIiMoGeTkVtdU8kIiybdUoCabSJzEZeJ6t369baJpG2dLACgxcioijF8gpXhGWz0eSUxFo0gaIRIgSXVgdQIuBqIyKiKMS6bLsIox56SvR3NSIEl3oCKKdi8EJEZJAdZdtFWTardT+nrsbq4DJ4eXxldX3Y95YIAZTVOG1ERGSQHVMEIi2btbI4nlNZOaWmdXpShNE5q3HkhYjIIK1Xrlv3H1e9UtZL66iHlqv0aMk5JVYUx3Miq6bU9ExPijI6ZyUulSYiMkhrvZNgZibyqq1wUrpKf2BqAXqlxFtaVI3MTeI2sjw+2jo3dmCdFwYvRBQDcqeiNEUQTiw6D6VlsuHIHSoAoWqCuCGQMutvMFoU0M11Xhi8EBFFQekKV42VheTa/BLGP1yBWp+2Ka1wmy0G/w6I/VV6uE43IyUeM4pycU1BtiMDmWhEUxTQSUEgi9QREcWInH8S2tmqkRN5t1XXw+v1mNqxrKjYpzlwkdui9rtY1wRRGjVqaG7Fs1sP4tmtB4UePbBCNAm4sa5zEysceSEiMkHwFe6+uias2FQd8T7pSd1x8sz5zQ3N2Pvmjm9GgcwWi32KIuV2BPNAzLwNK0SanozFlhCxoKf/tmy10a9+9SuMGzcOycnJSE9P13QfSZLw4IMPIicnB0lJSSgtLcW+ffusaiIRkWmCV92MH9xH032CAxcgusJ2cs0Zq4SurLJiJVOkpeehzK6hIyqtK5gAWL66TBSWTRu1trbi+9//PkpKSvDMM89ous9vfvMbPPbYY3jhhReQn5+PBx54AJMmTcKePXuQmOjc9ehE1LVEqvWhJJppGr0dv17BUxJWJYLqKZoWizL7IlGanswOSrgOHbVy8/SaZcHL4sWLAQDPP/+8puMlScKyZctw//33Y/r06QCAF198EVlZWXj99ddx4403WtVUIiJTqRWSi8Rop2xVtdTQompWbvhnpGhaLKvE2p38qlQUcP2eWldvwhiOMAm7Bw4cQG1tLUpLS9tvS0tLw9ixY1FZWakYvLS0tKClpaX9Z5/PZ3lbiYgiUbpSDs1zUaK3UzZaLTU4uIpUsdfqDf+MjFjFqkqsKMuOQxNwu8ImjOEIU2G3trYWAJCVldXh9qysrPbfhVNWVoa0tLT2f3l5eZa2k4hIq8nDc7Dl3ol4Zc7lWH5jEV6ZczkenzlK030zeyToeq5IVVUBID2pG7JTOz5udloiVt1yKVaFqdibkRKPlTedv2q3esM/tdyOUEaqxBrN04n15pt6dIVNGMPRNfJy3333YenSparHfPbZZxg6dGhUjdJj4cKFWLBgQfvPPp+PAQwRCSP0SnnrvuPa7qgz11LLnkcPf29k2GkHINAJTr4kC3/e+W80nW0DANQ3t2LJuj3wegOBWCw2/NOy9NxImX2jIyeij2x0hU0Yw9EVvNxzzz2YPXu26jEXXnihoYZkZ2cDAOrq6pCTc/6NVFdXh6KiIsX7JSQkICFB3xUKEZFdjje3RD5Ix3HBIiV1yp10pCqswYLzJmK14V9wbseGPbV4reooGprPT7WF/j2RRMrTuav0IgzMTA6bx2LH5pt6dIVNGMPRFbz06dMHffpoWwKoV35+PrKzs7Fx48b2YMXn8+HDDz/EnXfeaclzEhHFmtWdjZ6dnrVsIxA8uvD+z6+ybMfkUPKIVcmg3vjF1ALDibKRRk4A4NENX7TfFjoaI/rIhpW7WIvMsoTdw4cPo6GhAYcPH0ZbWxuqqqoAAIMHD0aPHj0AAEOHDkVZWRm+853vwOPx4K677sJDDz2EIUOGtC+Vzs3NxYwZM6xqpnb+NuDQB8CpOiA5E/B4gOavgIR04PO3gINbgLjuwMgbgJK5gDfu/PE9soC8scCRD8//PGBc4Bj5cZtqgFPHgNMNgcdOSgdS+gDNxwO3eb3AgAlA/rcC9wvXJkkCDv0DOHE48HsPgPQBwMBvBZ5Pfv6UPsC5FmD3n4Gzp4DUbKDfGODsCSC5d+Dvam4Amo4CaXnAgPGAxwucPt7xb5f/roNbgKo1QOORwPFFNwMDx3f+ewHgwD+AQ1sC3xr53wIGTlC+/etWYP39QMO/gIwLgWseAuKTArd/tDpwOwD0uwxI63f+OeRzktIncE6ajwXam9gL+HJH4JiMC4HL5gT+X+mxgl+fU3WB+3/1GXDyUOC89hoEfPYa0NIE9MwB8oqB1H7nX+umGsBXA9R9CrSeAQaUAMU/BrrFa3ufBb9PgI5/t+QHElIDz5GWF3iNld4bwY8jP8aJg0CvgcDoHwJHPz5/XL8xwI5nz/9+1Czgny8EPWc6AAk4cwI4tifwt2ddAhTdcv41b6oJnO+k3sCZ+sB76nT9+ffWmZOBN6f8OsttjnQegl/b0HMj+7oV+PBJ4Mg2oHsKMOIHQFw35fducHtT+gRex9DH/boV+OBx4OPngRYfkNwL6NEPkFqB7klA7ujA3yJJgffwl/8M3D5gHFD8H4HHOvAP4OBm4OS/MTa1H+5O+gpxrSfRz3McF3pqIEFCHXqjvK0YNcjEkR6FHTubcN8TXi+QMxrY9y5Q+08gPg0YfBWQmou45D4oObYr8DoCwKnLgB7Zgb//VB3Q/BX8yZl48+1aeHAh4uDHrXHvYqCnFlmeBkACmj2JOCUlQkIcDklZeLHxWuyorsOzF32IQ//ciGZPPPb4B6BeSkeGpwknpGQUef+FK/tlIu7vfwNyRwXORfDnTe29r/QeaKpBXFMdSs6eAOAFUr8FYALgx/nvDb8fSM4AevQFUvoG3quHPwAk4PPuwzGwaS8u8/rwFQKFz/rAh2NIx8f+i3CZ93OUePcAErBNGgo0efG3NRXIKL0MxRMmYeSRNfhlt+04LPXFy22luNS7H31xEseQju3+QIpEiXc3Sj55Ffj8a+CCyzu+7sHfa/2LA58v+fum/2WB74xw7+Vwn4Uw391x/YuxfGwiZr4bh274Gr/o9jIGeupwUMrCw1/fhCLvfizJb0DcpsrAdzmkwON5cL5PAYB/bQY+WQOcPAyk5wEjZwZ+F+57PNLnMAYsq7A7e/ZsvPDCC51u37RpE6688srAk3s8eO6559qnoiRJwqJFi/DUU0/h5MmTmDBhAv77v/8bF110kebntaTC7p43gfJ7Ad+X2u8T3wNoPXX+Z4838IGSpeYCw/83sOvP+h43qRcw7bHA/+tqk94Fm9EKeb6kDKCtBWht7nhYfI/AcaG3e7sD/jArMnrlB4KH4HMZ/Bxyx2qkjcGMvj5A59c69DnH/V/g2iWdfxXufZaaC0xeCvz7I6ByhcrjQv29kZoL5BQBX5SrP0ZUDLzHkjKAacuBguvP3xbuPIR7beVzI9/33QcCQYbWNii9TsGP++4DwAeP6fubQnVLAL7WNwV0JikbSdMeCbTByPePDk1SIpLRgjiP+nnzS4HYx3BWh8cLlMwL/94PFelvVvre0KlN8qj+3fKoU/vP35wDWYPUA0loQZInzHdVt0Tga42jMaHvZZmO177NmwCvv6Vje6Hh9YpPCQRIWtqa1CvwiGeCkn+V2m4AN2Y0M3jZ8ybw6q2IbcdPrjfupx2/xBXfZ7EOOm3yg5fOd9SaP2/ffC3/4MVAcBdtkBH62BdPAfb+zcTHNNCGcf9XX0BmgNwDeCL0ctI3PWHUKamh7/1QMfzODQ1GOv0e6sGL1nOnjSfwXpaDAAPnoVN7YcLrFVHQ5zDKAIbBi1nBi78NWDbcsise6sI8ccD/qw0Mo/N9Fhg6/2kV8FihzvPgCUz1nKq1cETJLp5Ar+i2vyv4vR+qq38WUvsBd30a+H9HnQdPYATmrk+jmkISYm8jVzj0gYPePOQoUlsg9wTg+wwAfEcD50P3eZCApi/d18EDACR3/l3B7/1QXf2z4DsaOAeOOw/S+bbHiDAVdoV0qs7uFpCbtSdT8n0G4Pz5IPdTeq35WXD2OYhh2xm8qOmRFfkYIqN6DQz8l++zAPl8kPspvdb8LDj7HMSw7Zw2UjNgXGAeLwYpT9TFeOLOL9Xm+yww13/ZHAPnwQP0zA2sZHEdjzv/ruD3fqiu/lmQl0077jx4zrc9Rlz4yTCRNy6wBAyAmG8iEdtEmpTMPZ+wqPo+M/oaO+y9MfnhwPnQ9Xn75pgpSwNLcE3lAS6+zuTHNKBkHqx+LSUpxuvZgt/7oTp8FsQn4fyKow63S+FvV+cJfA68cQ7oe4J90z657THC4CWSgusDS8BSde4aGt+j48+hV1Cp/QJLBlNz9T1uUkZgWekPXtLZplh/AEKeLykjUE8gVHyP8Ld7u4d/2F75ylejSRnf1CEw2MZgRl8fIMLVsif8UlGl91lqbuC1HvfTyFfhau+N1H6BztjSK3kD7zG5zfISS6XzEO61Tc09vzzz2iWBc6SnDUrnIrVf4HFnvvLNY0apm4HtS+Q2XLvkm/Nh4H2o0SkkQlewaJQnLvIyaSDoPaDyNyt9b+hvVFT3bkRPnEXn7yo/EPZ2RfLrHbzUWG/f081g+f/4Htrvm5TxTc2lIMGfwxjiUmmtWGGXFXZZYZcVdk2osIvUfoHP99mTwIlDQP3+wPOk5gLDrg9UNw1pQ9vXX+PzD9/BmYYj+LqxFh99/i9IkgfbpcG41rsThZ4DaEQKNvsvwVdSbxxHGh663IMB3mP4y85/44OzA1GLQKfTFyfR2+NDvZSKOmTgSI9CbP7ZFYjb8XSgLb6awHnqlgwkpga+e4I/L/L57ZaEA/GDsXzbyQ4VduXKRJ/4B6LQexCAB98eOxb5U+YbrrCLprrA9xO8Hb43tm96A9sq3oDHI+GE1AP1UhqOIR1e+DHW8zmuG5GNwWMmB/6e08fPfyecPn7+PXC4MvD6eADklQTeL/J7JPTzEPR52e1LwrS3AqvBSry78V3vZqR4WvCR/yK80DYZfngx1rsHv7m0EXm9ki2rsIv+xee/s0K/M69eHGiv/N0qeIVd1nmxInghIrJBpI0TlSy/sQjTi/q1718EhN9p+olbLtW8wWGoNr+ECUsrIu6rs+XeiabvuCw/t9rO01Y9NwC8UXUU89dWRTxOfh0oMtZ5ISJyATnw0Bu4AOc3dpR3ms5O6zg1kJ2WGFXgAgQ2T1w0rQCAcrbWomkFlgQPenZ7tkJX3c1ZFFwqTUQkILXdkNWE20VYz07TesnBUejoUHbI7sxms3u35666m7MoGLwQEQko0shCOGqjHXFeD0oG9TapdR1ZGRwpsXvkQx51uvPlnZ12ILN61IkYvBARCcnIiIHVox1qQoOjNr+Eyup6y4IZEUY+7Bp1IgYvRERC0jpi8MDUYcjsmRCT0Q6twiUZ55jcoYsy8mHHqBNxtRERkZDsXMkTDTnJOLTNZqxuUno+qwMlig09/TdHXoiIBCTKyIIeaknGEgLtXvzWHlxTkG1auzny0TUxeCEiEpTTcir0LF+ONnm4zS8xYOnCGLwQEQnMSSMLsVq+HGmqiIGN+zF4ISISnJXLnM0Ui+XLSjk1tY1ncefLO/Hjb+fjzU9qmAPjcgxeiIhIk0gjGlYsXw5+zsweCfjlm7sVc2oA4MnNBzr9Tg5szE4WJvsweCEiooi0rOoxO8nY6L5OoaxKFib7cG8jIiJSpbTHkjyiUb6rpv02s/ZSimZfp3C07HUkF9Z7o+ooKqvr0eZ3VSURV+HICxERKTKy/DnaJGOj+zppoZQszHoxzsLghYiIFGld/rytuh5er6dDsGI0ydjIvk5ahUsWjpQEzFwZ8TB4ISIiRVqXNc9dsxMnz5xr/zmaUQurdoLOCZMsbEdhPYoec16IiEiR1mXNwYELED4fxuznDKYlrnhgaudkYT2F9UgcDF6IiEiRvPxZ75iDPJKx+K09uhNfIz2nB4FRlD/cPhbLbyzCA1OHQctT9EqJ73RbrArrkbkYvBARkSJ5+TMAQwGMkVELtecMXnI9fkgmphf1Q2bPBE2PGy4AiUVhPTIfgxciIlKltPw5Pam7pvsbGbXQs+Q6mgBE6yiPnsJ6ZD0m7BIRUUThlj/7JQk3P/1hxPsaHbXQuuQ6msq+Tty9mxi8EBGRRqF7LLX5JdO3A4j0nErHRBOAOG33bgI8kiS5qoSgz+dDWloaGhsbkZqaandziIhcJXR/oxPNrZi7ZieA8EFDLGukRFtojrtR20tP/83ghYiINFEKDq4vzAm7k/ONl12AgZnJMQ0EGIA4F4MXBi9ERKZSqkIrhwUrbxqFXikJONZ0FgePN+OV7YdR62tpP46l9ikSPf03VxsREZGqSFVoAWDJus9QnJ+BhG5eLNuwr0PgAkRXtI4oFIMXIiJSpXl/o3/VRwxyjBStIwrF4IWIiFRprdNSWV3PUvsUEwxeiIhIlfY6LdpGVFhqn6LF4IWIiFRprULbzautS2GpfYoWgxciIlKlZa+h6wtzsHzjPtXHYal9MguDFyIiikhtr6GVN43Cm5/URJw0ksBS+2QOy7YH+NWvfoV169ahqqoK8fHxOHnyZMT7zJ49Gy+88EKH2yZNmoTy8nKLWklERFop7TUUaTWS7O7SIazzQqawLHhpbW3F97//fZSUlOCZZ57RfL/Jkyfjueeea/85IUHbVudERGS9cHsNaU3AHZiZYkWTqAuyLHhZvHgxAOD555/Xdb+EhARkZ2db0CIiIrKC1gRcJuqSWYTLeXnvvffQt29fXHzxxbjzzjtRX1+venxLSwt8Pl+Hf0REFDtaVyMxUZfMIlTwMnnyZLz44ovYuHEjli5divfffx9TpkxBW1ub4n3KysqQlpbW/i8vLy+GLSYiIi2rkZioS2bSFbzcd9998Hg8qv8+//xzw4258cYbcf3112PEiBGYMWMG3n77bXz00Ud47733FO+zcOFCNDY2tv87cuSI4ecnIiJj1FYjPXHLpUzUJVPpynm55557MHv2bNVjLrzwwmja0+mxMjMzsX//flx99dVhj0lISGBSLxGRAJRWI3HEhcymK3jp06cP+vTpY1VbOvn3v/+N+vp65OQwYicicoJwq5GIzGZZzsvhw4dRVVWFw4cPo62tDVVVVaiqqsKpU6fajxk6dChee+01AMCpU6fw85//HNu2bcPBgwexceNGTJ8+HYMHD8akSZOsaiYRERE5jGVLpR988MEOBedGjRoFANi0aROuvPJKAMDevXvR2NgIAIiLi8P//M//4IUXXsDJkyeRm5uLa6+9FkuWLOG0EBEREbXzSJKkbRtQh/D5fEhLS0NjYyNSU1Ptbg4RERFpoKf/FmqpNBEREVEkDF6IiIjIURi8EBERkaMweCEiIiJHYfBCREREjsLghYiIiByFwQsRERE5CoMXIiIichQGL0REROQoDF6IiIjIURi8EBERkaMweCEiIiJHYfBCREREjsLghYiIiBylm90NICIi8bT5JWw/0IBjTWfRt2ciivMzEOf12N0sIgAMXoiIKET5rhosfmsPahrPtt+Wk5aIRdMKMHl4jo0tIwrgtBEREbUr31WDO1/e2SFwAYDaxrO48+WdKN9VY1PLiM5j8EJERAACU0WL39oDKczv5NsWv7UHbf5wRxDFDoMXIiICAGw/0NBpxCWYBKCm8Sy2H2iIXaOIwmDwQkREAIBjTcqBi5HjiKzC4IWIiAAAfXsmmnockVUYvBAREQCgOD8DOWmJUFoQ7UFg1VFxfkYsm0XUCYMXIiICAMR5PVg0rQAAOgUw8s+LphWw3gvZjsELERG1mzw8B0/ccimy0zpODWWnJeKJWy5lnRcSAovUERFRB5OH5+CagmxW2CVhMXghIqJO4rwelAzqbXcziMLitBERERE5CoMXIiIichQGL0REROQoDF6IiIjIURi8EBERkaMweCEiIiJHYfBCREREjsLghYiIiByFwQsRERE5iusq7EqSBADw+Xw2t4SIiIi0kvttuR9X47rgpampCQCQl5dnc0uIiIhIr6amJqSlpake45G0hDgO4vf78eWXX6Jnz57weMzZRMzn8yEvLw9HjhxBamqqKY/pJjw/6nh+1PH8RMZzpI7nR51Tzo8kSWhqakJubi68XvWsFteNvHi9XvTv39+Sx05NTRX6hbcbz486nh91PD+R8Ryp4/lR54TzE2nERcaEXSIiInIUBi9ERETkKAxeNEhISMCiRYuQkJBgd1OExPOjjudHHc9PZDxH6nh+1Lnx/LguYZeIiIjcjSMvRERE5CgMXoiIiMhRGLwQERGRozB4ISIiIkdh8BLBypUrMXDgQCQmJmLs2LHYvn273U0SxubNmzFt2jTk5ubC4/Hg9ddft7tJQikrK8Nll12Gnj17om/fvpgxYwb27t1rd7OE8cQTT2DkyJHthbNKSkrw97//3e5mCevhhx+Gx+PBXXfdZXdThPDLX/4SHo+nw7+hQ4fa3SyhHD16FLfccgt69+6NpKQkjBgxAh9//LHdzTIFgxcVf/zjH7FgwQIsWrQIO3fuRGFhISZNmoRjx47Z3TQhNDc3o7CwECtXrrS7KUJ6//33MXfuXGzbtg3r16/HuXPncO2116K5udnupgmhf//+ePjhh7Fjxw58/PHHmDhxIqZPn47du3fb3TThfPTRR3jyyScxcuRIu5silEsuuQQ1NTXt/7Zs2WJ3k4Rx4sQJjB8/Ht27d8ff//537NmzB7/73e/Qq1cvu5tmDokUFRcXS3Pnzm3/ua2tTcrNzZXKyspsbJWYAEivvfaa3c0Q2rFjxyQA0vvvv293U4TVq1cv6emnn7a7GUJpamqShgwZIq1fv1664oorpPnz59vdJCEsWrRIKiwstLsZwrr33nulCRMm2N0My3DkRUFrayt27NiB0tLS9tu8Xi9KS0tRWVlpY8vIqRobGwEAGRkZNrdEPG1tbVi7di2am5tRUlJid3OEMnfuXEydOrXDdxEF7Nu3D7m5ubjwwgtx88034/Dhw3Y3SRhvvvkmxowZg+9///vo27cvRo0ahdWrV9vdLNMweFFw/PhxtLW1ISsrq8PtWVlZqK2ttalV5FR+vx933XUXxo8fj+HDh9vdHGF8+umn6NGjBxISEnDHHXfgtddeQ0FBgd3NEsbatWuxc+dOlJWV2d0U4YwdOxbPP/88ysvL8cQTT+DAgQP41re+haamJrubJoR//etfeOKJJzBkyBC88847uPPOO/HTn/4UL7zwgt1NM4XrdpUmEtHcuXOxa9cuzsmHuPjii1FVVYXGxkb8+c9/xqxZs/D+++8zgAFw5MgRzJ8/H+vXr0diYqLdzRHOlClT2v9/5MiRGDt2LAYMGIBXX30Vt99+u40tE4Pf78eYMWPw61//GgAwatQo7Nq1C6tWrcKsWbNsbl30OPKiIDMzE3Fxcairq+twe11dHbKzs21qFTnRvHnz8Pbbb2PTpk3o37+/3c0RSnx8PAYPHozRo0ejrKwMhYWFWL58ud3NEsKOHTtw7NgxXHrppejWrRu6deuG999/H4899hi6deuGtrY2u5solPT0dFx00UXYv3+/3U0RQk5OTqeLgGHDhrlmao3Bi4L4+HiMHj0aGzdubL/N7/dj48aNnJMnTSRJwrx58/Daa6+hoqIC+fn5djdJeH6/Hy0tLXY3QwhXX301Pv30U1RVVbX/GzNmDG6++WZUVVUhLi7O7iYK5dSpU6iurkZOTo7dTRHC+PHjO5Vm+OKLLzBgwACbWmQuThupWLBgAWbNmoUxY8aguLgYy5YtQ3NzM2677Ta7myaEU6dOdbjKOXDgAKqqqpCRkYELLrjAxpaJYe7cuVizZg3eeOMN9OzZsz1XKi0tDUlJSTa3zn4LFy7ElClTcMEFF6CpqQlr1qzBe++9h3feecfupgmhZ8+enfKjUlJS0Lt3b+ZNAfjZz36GadOmYcCAAfjyyy+xaNEixMXFYebMmXY3TQh33303xo0bh1//+tf4wQ9+gO3bt+Opp57CU089ZXfTzGH3cifRPf7449IFF1wgxcfHS8XFxdK2bdvsbpIwNm3aJAHo9G/WrFl2N00I4c4NAOm5556zu2lC+OEPfygNGDBAio+Pl/r06SNdffXV0rvvvmt3s4TGpdLn3XDDDVJOTo4UHx8v9evXT7rhhhuk/fv3290sobz11lvS8OHDpYSEBGno0KHSU089ZXeTTOORJEmyKW4iIiIi0o05L0REROQoDF6IiIjIURi8EBERkaMweCEiIiJHYfBCREREjsLghYiIiByFwQsRERE5CoMXIiIichQGL0REROQoDF6IiIjIURi8EBERkaMweCEiIiJH+f8+ScjgUOe+LQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x_samples, y_samples)\n",
    "plt.scatter(x_samples, network_forward(x_samples, weight_matrices, bias_vectors, activation_functions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_forward(y_guess, y_ref):\n",
    "    delta = y_guess - y_ref\n",
    "    return 0.5 * np.mean(delta**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8150125501544493"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_forward(\n",
    "    network_forward(x_samples, weight_matrices, bias_vectors, activation_functions),\n",
    "    y_samples,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_backward(y_guess, y_ref):\n",
    "    delta = y_guess - y_ref\n",
    "    N = delta.size\n",
    "    return delta / N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (1421862268.py, line 37)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Cell \u001b[0;32mIn[13], line 37\u001b[0;36m\u001b[0m\n\u001b[0;31m    weight_grad =\u001b[0m\n\u001b[0m                 ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "def network_forward_and_backward(x, y_ref, weights, biases, activations, activations_derivatives):\n",
    "    a = x\n",
    "\n",
    "    # Store the intermediate activated states for the backward pass\n",
    "    layer_states = [a, ]\n",
    "\n",
    "    for W, b, f in zip(weights, biases, activations):\n",
    "        a = a @ W\n",
    "        a = a + b\n",
    "        a = f(a)\n",
    "        layer_states.append(a)\n",
    "    \n",
    "    y = a\n",
    "\n",
    "    loss = loss_forward(y, y_ref)\n",
    "\n",
    "    current_cotangent = loss_backward(y, y_ref)\n",
    "\n",
    "    weight_gradients = []\n",
    "    bias_gradients = []\n",
    "\n",
    "    for W, f_prime, a_current, a_prev in zip(\n",
    "        reversed(weights),\n",
    "        reversed(activations_derivatives),\n",
    "        reversed(layer_states[1:]),\n",
    "        reversed(layer_states[:-1]),\n",
    "    ):\n",
    "        activated_state_cotangent = current_cotangent\n",
    "        plus_bias_state_cotangent = activated_state_cotangent * f_prime(a_current)\n",
    "\n",
    "        bias_grad = np.sum(plus_bias_state_cotangent, axis=0)\n",
    "\n",
    "        state_cotangent = plus_bias_state_cotangent\n",
    "\n",
    "        prev_activated_state_cotangent = state_cotangent @ W.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "265371ff1b98b9f4eaa16d44fb1eb5bb5e02f4557e1c68186d1d500959ccd159"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
